Quest: Real Data Pipeline
Layer: Forge
Priority: P92
Status: active
Tasks: 5 total (0 done, 5 open)
Vision
Ensure all analyses use REAL Allen Institute datasets (SEA-AD, ABC Atlas, Allen Brain Cell Atlas) — not simulated/hallucinated data. Build data ingestion pipelines that download, cache, and validate real datasets. Analyses must cite specific dataset versions and cell counts. Forge tools must be invoked during the analysis loop with real inputs, not just registered as provenance markers.
This quest is not just about fetching files. It should produce analysis-ready
evidence bundles that combine real expression data, pathway/network context,
and literature support into mechanistic explanations that can be cited by
debates, notebooks, pricing, and downstream validation tasks.
Tasks
☐ [Forge] Download and cache SEA-AD transcriptomic datasets from Allen Brain Cell Atlas (P5)
☐ [Forge] Integrate real Allen data into the analysis/debate pipeline (P5)
☐ [Forge] Build data validation layer — verify analyses cite real datasets (P4)
☐ [Forge] Ensure Forge tools are invoked during analysis execution, not just registered (P4)
☐ [Forge] Add ABC Atlas (Allen Brain Cell Atlas) and MERFISH spatial datasets (P3)
☐ [Forge] Generate mechanistic evidence bundles for real-data analyses (P5)
☐ [Forge] Route debate and pricing updates through fresh real-data evidence bundles (P5)
Success Criteria
☐ All tasks completed and verified
☐ Integration tested end-to-end
☐ Metrics visible on Senate/Quests dashboards
☐ Design supports future scaling to external compute
☐ All analyses spawned by the system cite real datasets (not simulated placeholders). >95% of analysis outputs include dataset version and cell count. Forge tool invocations during analysis are logged and validated — no synthetic fallbacks.
☐ Debate prompts and exchange repricing jobs can consume those bundles
without fabricating values or falling back to synthetic placeholders
Architecture Notes
This quest is designed with a local-first, cloud-ready philosophy:
- All implementations must work on the current single VM
- All interfaces must support future migration to containers/cloud
- Resource limits are configurable, not hardcoded
- Executor/sandbox abstractions allow swapping implementations
- Evidence pipelines should emit structured JSON/CSV/markdown outputs with
provenance, so downstream consumers can compare runs and detect stale inputs
Work Log
_No entries yet._