[Forge] Ensure Forge tools are invoked during analysis execution, not just registered
Quest: Real Data Pipeline
Priority: P4
Status: open
Goal
Audit and fix the analysis pipeline to ensure Forge tools (PubMed, Allen, STRING, ClinVar, etc.) are actively called during each analysis, with results feeding into debate rounds. Currently tools are mostly called by enrichment scripts, not the main debate loop.
The stronger target is an evidence-bundle pattern: analyses should gather tool
results into a structured, reviewable package that debates, notebooks, and
exchange updates can all consume, rather than reissuing ad hoc queries in
every downstream step.
Acceptance Criteria
☑ Each analysis invokes at least 3 Forge tools during execution
☑ Tool results are included in debate persona prompts as evidence
☑ tool_calls table shows tool invocations linked to analysis IDs
☐ Analysis quality score correlates with number of tools used
☐ Dashboard shows tool usage per analysis
☑ At least one production analysis has a persisted mechanistic evidence bundle
assembled from tool outputs and linked to the analysis
Approach
Audit scidex_orchestrator.py — trace where tools are (or aren't) called in the debate loop
Audit engine.py — understand the tool execution model
Modify debate persona prompts to require tool-backed evidence
Add tool invocation step before each debate round
Log tool_calls with analysis_id foreign key
Add tool usage metrics to /analyses/ pageDependencies
_Identify during implementation._
Dependents
_Identify during implementation._
Work Log
2026-04-20 22:15 UTC — Slot minimax:61
- Audit: scidex/agora/scidex_orchestrator.py, scidex/forge/tools.py, migrations/add_tool_calls.py
- Key finding: execute_tool() in orchestrator did NOT log to tool_calls table — tools
executed by debate personas were invisible to Forge provenance. @log_tool_call in
tools.py already logs but analysis_id was hardcoded to None.
- Key finding: research_topic() is called 3x per debate (Theorist, Skeptic, Expert)
and results fed into literature_section — this is the primary tool invocation path.
- Fix 1: execute_tool() now logs to tool_calls with analysis_id from self.current_analysis_id
- Fix 2: run_debate() initializes self.evidence_bundle and appends research_topic()
results at each of the 3 literature-fetch phases
- Fix 3: save_analysis() persists evidence_bundle to analyses/<id>/evidence_bundle.json
with output summaries (not full output, to avoid bloat)
- Committed: 5789e4d4a — [Forge] Log Forge tool calls with analysis_id; add evidence bundle persistence
- Remaining: tool usage on /analyses/ dashboard, correlation check (P5 scope)