validator LLM call crashed: RuntimeError("All LLM providers failed. Last error: CLI harness codex_cli returned exit 1: Error: No such file or directory (os error 2)\n. Tried: ['minimax', 'glm', 'claude_cli', 'codex_cli']. Check API keys and provider availability.")
Group and triage failed tool calls so recurring Forge failures become fixes or targeted follow-up tasks. This improves reliability for debates, analyses, and autonomous research loops.
tool_calls with status = error and group by skill_id plus normalized error_message.q-cc0888c0004a - Agent Ecosystem quest66bd4bd4-7c04-41c2-b332-74b1a9baf7dc.AGENTS.md, CLAUDE.md, this task spec, quest spec quest_agent_ecosystem_spec.md, and alignment-feedback-loops.md.tool_calls has 389 rows with status='error' and 27,040 rows with status='success'.skill_id and error pattern, apply narrow local compatibility fixes for deterministic argument-shape failures, and document remaining caller/upstream causes plus before/after untriaged counts.docs/code_health/tool_call_failure_triage_2026-04-21.md with the latest 50 failed tool_calls grouped by skill_id and error pattern.scidex/forge/tools.py: query aliases and empty input handling for PubMed, Semantic Scholar, ClinicalTrials, research topic, Open Targets, KEGG, AlphaFold, paper figures, and paper corpus ingest.skills.input_schema and skills.example_input.python3 -m py_compile scidex/forge/tools.py; targeted Python smoke checks for empty/alias calls returned structured empty results without increasing status='error' rows.9d3be8ecb is already pushed and targeted to the expected three task files.research_topic(query=..., max_papers="0"); patched research_topic to accept max_papers/max_results and coerce numeric string limits.python3 -m py_compile scidex/forge/tools.py; targeted smoke checks including research_topic(query="APOE glia", max_papers="0") returned an empty evidence brief without raising argument-contract errors.status='error', 27,295 rows with status='success'; original required 50-row batch remains triaged and the extra discovered mode is documented in the code-health addendum.max_papers="0" avoided argument errors but still allowed ClinicalTrials.gov to return default results.research_topic so an evidence limit of zero returns empty PubMed, Semantic Scholar, and ClinicalTrials lists without provider calls.python3 -m py_compile scidex/forge/tools.py; research_topic(query="APOE glia", max_papers="0") returned 0 total evidence and no provider rows.origin/main; it remains at 863577266, so the task branch is still based on the current main snapshot.python3 -m py_compile scidex/forge/tools.py.research_topic, and research_topic(query="APOE glia", max_papers="0"); zero-limit research returned 0 PubMed papers, 0 Semantic Scholar papers, and 0 ClinicalTrials rows.psql prompted for a password in this harness, so live counts were checked through scidex.core.database.get_db(): 390 error rows and 27,404 success rows. The original 50-row triage batch remains documented, leaving 339 untriaged rows by the report accounting.data/papers/*.json, deleted scripts/cache_paper_fulltext.py, and an api.py quality-gate regression.origin/main (19cbede2b) and limited the corrected branch content to the intended Forge triage files: scidex/forge/tools.py, docs/code_health/tool_call_failure_triage_2026-04-21.md, and this spec.api.py, data/papers/*.json, scripts/cache_paper_fulltext.py, and unrelated specs from the corrected task commit.python3 -m py_compile scidex/forge/tools.py; signature checks for all alias parameters passed; smoke checks for empty calls and research_topic(query="APOE glia", max_papers="0") passed without provider calls.scidex.core.database.get_db(): 390 error rows and 27,618 success rows. The original 50-row batch plus 1 retry addendum row leaves 339 untriaged failed calls by report accounting.main, including unrelated clinical-trial script deletions and unrelated spec edits.main (3b914af08) plus only the Forge triage deliverable. The code-health triage report already exists on current main, so the repaired diff is limited to scidex/forge/tools.py and this spec.api.py, data/papers/*.json, scripts/cache_paper_fulltext.py, backfill_figures.py, or the quality-gate spec.python3 -m py_compile scidex/forge/tools.py; targeted smoke checks for alias/empty inputs passed, including zero-limit research_topic returning no PubMed, Semantic Scholar, or ClinicalTrials rows.0cacff47d87d0c33d (squash-merge of task 66bd4bd4) committed all prior work (tools.py fixes, code-health report, spec work-log) to origin/main.gene_symbol alias mismatches from chembl_drug_targets, string_enrichment, and methbase_disease_methylation.gene_symbol alias support:chembl_drug_targets(target_gene=None, gene_symbol=None, max_results=10) — accepts gene_symbol as upstream caller conventionstring_enrichment(gene_symbols=None, gene_symbol=None, species=9606) — accepts single-gene gene_symbol kwargmethbase_disease_methylation(disease, gene=None, gene_symbol=None, max_results=10) — accepts gene_symbol alias for gene
[] on empty calls.chembl_drug_targets(gene_symbol='APOE') → 10 items; string_enrichment(gene_symbol='APOE') → 20 items; methbase_disease_methylation('Alzheimer', gene_symbol='APOE') → 10 items; all without argument errors.python3 -m py_compile scidex/forge/tools.py → ✓ Syntax OK.gene_symbol patterns. These 9 will no longer recur. Remaining 386 errors include older all-time patterns (pubmed_search query=missing, paper_corpus_ingest type errors, alphafold_structure alias, etc.) already documented in the code-health report.7008b540search_trials() with status kwarg (9 errors) — added status=None paramsemantic_scholar_search() with limit kwarg (5 errors) — added limit=None alias for max_resultsstring_enrichment() with max_results kwarg (3 errors) — added max_results=None param with truncationpaper_corpus_search() with max_results kwarg (3 errors) — added max_results=None as alias for per_pageget_disease_info() with disease_term kwarg (3 errors) — added disease_term=None alias
python3 -m py_compile scidex/forge/tools.py → ✓.7106f7bfe to branch orchestra/task/7008b540-triage-50-failed-tool-calls-by-skill-and.docs/code_health/tool_call_failure_triage_2026-04-21.md.8151665ce), which already contains all prior iteration fixes on origin/orchestra/task/7008b540-triage-50-failed-tool-calls-by-skill-and.string_protein_interactions(gene_symbol=...) already accepts the gene_symbol alias and gene_symbols=None default (iteration 3 fix, 8151665ce).pubmed_search(terms=...) already accepts the terms kwarg (iteration 2 fix, af6b149c8).8151665ce; git push origin HEAD reports "Everything up-to-date".skills.input_schema (upstream registry coverage issue).origin/main; verified clean diff (only scidex/forge/tools.py changed, 9 insertions, 3 deletions).pubmed_search(term='cancer'), pubmed_search(terms='cancer') → OKsemantic_scholar_search('cancer', limit=5) → OKsearch_trials('cancer', status='RECRUITING') → OKstring_enrichment(gene_symbol='TP53', max_results=3) → OKpaper_corpus_search('cancer', max_results=3) → OKget_disease_info(disease_term='cancer') → OKstring_protein_interactions(gene_symbol='TP53') → OKpaper_figures(pmid='20441996') → OK
python3 -m py_compile scidex/forge/tools.py → ✓.push_resource_context) are different failure modes requiring separate triage workflows.status='error' rows. Confirmed 2 argument-alias gaps not yet on origin/main:pubmed_search(terms=...) — missing terms kwarg; added terms=None param and query = query or term or search_query or terms.string_protein_interactions(gene_symbol=...) — missing gene_symbol kwarg; added full alias set (gene_symbol=None, max_results=None, limit=None) with empty-call guard and max_results/limit cap.
pubmed_search(terms='cancer') → OK; string_protein_interactions(gene_symbol='TP53') → OK; string_protein_interactions() → []; string_protein_interactions('TP53', limit=3) → capped list.python3 -m py_compile scidex/forge/tools.py → ✓.5d1ede677 (disgenet_disease_genes + expression_atlas aliases), cf179c497 (pubmed_search terms + string_protein_interactions gene_symbol/limit aliases), 94994384d (restored string_protein_interactions max_results/limit cap after rebase cleanup).origin/main and force-pushed to remote.skills.input_schema coverage (registry-level fix required, not tool-code patch).status='error' rows, 34,705 status='success' rows.gene_list was a required positional arg causing TypeError on no-arg probes → fixed by making gene_list=None with early return [].except Exception: pass in PMID lookup silently left PostgreSQL transaction in aborted state, causing cascade failures on subsequent queries → fixed by adding db.rollback() in both paper_figures and _save_paper_figures_to_db exception handlers.
enrichr_analyze() → []; enrichr_analyze([]) → []; paper_figures(pmid='20441996') → count 0 without error.python3 -m py_compile scidex/forge/tools.py → ✓ Syntax OK.status='error' rows, 34,735 status='success' rows; 0 errors since last fix (14:01 UTC).gene="CHRNA7" but function only accepted gene_symbol=. Added gene=None alias with gene_symbol = gene_symbol or gene.query= instead of gene_symbol=; function also had gene_symbol as required positional causing no-arg probe failures. Made gene_symbol=None optional, added query=None alias with early return on empty input.
allen_brain_expression(gene='CHRNA7') → OK; allen_brain_expression() → []; allen_cell_types(query='CHRNA7') → OK; allen_cell_types() → {"gene": None, ...} without TypeError.python3 -m py_compile scidex/forge/tools.py → ✓ Syntax OK.Verification pass — all acceptance criteria already satisfied by prior triage work.
status='error' rows, 35,495 status='success' rows.7008b540 and dd1d8112).pubmed_search(term="cancer") → OK; pubmed_search() → []; research_topic(query="cancer") → OK; research_topic() → empty brief; search_trials(query="cancer", status="RECRUITING") → OK; paper_figures(pmid="31883511") → OK; alphafold_structure(gene_symbol_or_uniprot="TP53") → OK; string_protein_interactions(gene_symbol="TP53") → OK; allen_brain_expression(gene="TP53") → OK; allen_cell_types(query="TP53") → OK; disgenet_disease_genes(disease_name="cancer") → OK; expression_atlas_differential("TP53", organism="Homo sapiens") → OK; reactome_pathways(gene_symbol="TP53", max_results=5) → OK; brainspan_expression("TP53", max_results=5) → OK; gtex_tissue_expression("TP53", dataset=10) → OK; chembl_drug_targets(gene_symbol="TP53") → OK; string_enrichment(gene_symbols=["TP53"], max_results=3) → OK; mgi_mouse_models(gene_symbol="APP") → OK; paper_corpus_search("cancer", max_results=3) → OK; get_gene_info() → {}; enrichr_analyze() → []; uniprot_protein_info() → {}.4a8ea5eb9); all 451 error rows are pre-fix historical artifacts.python3 -m py_compile scidex/forge/tools.py → ✓.origin/main (df6838cbd): kwarg aliases, empty-input guards, and transaction rollback for paper_figures.Fresh triage of 50 most-recent error rows (2026-04-26); 51 total rows today.
status='error' rows, 36,192 status='success' rows.search_type="name" to active definition at line 3332 — the last of 3 duplicate defs; prior iteration targeted line 3119 which was shadowed.per_page=None kwarg that overrides max_results.gene_symbol=None optional with early return {"gene": None, "tissues": [], "error": None} guard.max_results=None as alias for max_per_collection.
pubchem_compound(search_type='name') → OK; openalex_works(query='x', per_page=5) → OK; gtex_tissue_expression() → {"gene": None, ...} ✓; msigdb_gene_sets(gene_symbol='TP53', max_results=5) → OK.python3 -m py_compile scidex/forge/tools.py → ✓.2ec734e49. Triage report updated: docs/code_health/tool_call_failure_triage_2026-04-21.md (Iteration 15 Addendum).status='error' rows, 37,425 status='success' rows; latest error was 2026-04-26 16:22:15.835282-07:00.docs/code_health/tool_call_failure_triage_2026-04-26_b4e04fba.md covering the latest 50 failed rows grouped into 18 exact skill_id + error-message patterns. Batch accounting leaves 432 historical rows outside this report.openalex_works(query=None, per_page=None) now accepts no-query probes and string per_page.expression_atlas_differential(gene_symbol=None, ...) now accepts no-argument probes and string max_results.brainspan_expression(gene_symbol=None, max_results=None) now accepts empty probes and string limits.msigdb_gene_sets(gene_symbol=None, max_results=None) now accepts empty probes and string limits.
python3 -m py_compile scidex/forge/tools.py; targeted smoke checks for OpenAlex, Expression Atlas, BrainSpan, and MSigDB returned structured results and added 0 new error rows after the fixes.origin/main to clean stale local slot file.pubmed_search(term=...) ✓, search_trials(status=...) ✓, paper_corpus_search(max_results=...) ✓, msigdb_gene_sets(max_results=...) ✓, pubchem_compound(search_type=...) ✓.query was required positional but callers passed max_results only → made query="" optional; also added string-to-int coercion for max_results.status=None accepted but never forwarded to the API → added if status: params["postFilter.overallStatus"] = status.
paper_corpus_search(max_results=5) OK; paper_corpus_search() OK; search_trials('cancer', status='RECRUITING') OK; search_trials() OK.python3 -m py_compile scidex/forge/tools.py → ✓.docs/code_health/tool_call_failure_triage_2026-04-26_b4e04fba.md.Rebased on latest main (75320e461), verified all prior fixes, documented live status vs. stale errors.
git rebase origin/main completed cleanly (1 auto-resolved conflict on .orchestra-slot.json).paper_corpus_search(query="") — present ✓paper_corpus_search int-coercion — present ✓search_trials status wiring — present (new to this branch, not on main) ✓paper_corpus_ingest non-dict skip — present (not on main) ✓brainspan_expression(gene_symbol=None) — present ✓brainspan_expression int-coercion — present ✓expression_atlas_differential(gene_symbol=None) — present ✓expression_atlas_differential int-coercion — present ✓openalex_works(query=None) — present ✓openalex_works int-coercion — present (not on main) ✓msigdb_gene_sets(gene_symbol=None) — present (not on main) ✓
search_trials status wiring is new to this branch and not yet on main.paper_figures transaction abort is infra-level DB session issue, not function signature problem.python3 -m py_compile scidex/forge/tools.py → ✓.docs/code_health/tool_call_failure_triage_2026-04-26_b4e04fba.md with iteration 5 findings.origin/main (commit eb39051c3). Verified 2 commits already on remote branch tip 5f8fba4c4.git push reports "Everything up-to-date".paper_corpus_search(query=""), search_trials(status) wiring, brainspan_expression(gene_symbol=None), expression_atlas_differential(gene_symbol=None).msigdb_gene_sets(max_results=), 12x pubchem_compound(search_type=), 3x openalex_works(per_page=), 3x gtex_tissue_expression(gene_symbol), 2x paper_corpus_search(query) — all already have the kwarg in the current signature but the code was updated to accept them.max_results=None on msigdb_gene_sets, search_type='name' on pubchem_compound, per_page=None on openalex_works, gene_symbol=None on gtex_tissue_expression, query=str="" on paper_corpus_search — errors are stale pre-fix artifacts from callers using old signatures.python3 -m py_compile scidex/forge/tools.py → ✓.paper_corpus_search(max_results=5) OK, paper_corpus_search() OK, search_trials('cancer', status='RECRUITING') OK, brainspan_expression() returns {'gene': None, ...}, expression_atlas_differential() returns {'gene': None, ...} — all PASS.docs/code_health/tool_call_failure_triage_2026-04-26_b4e04fba.md with iteration 2 findings.status='error' rows; 202 from last 7 days (stale pre-fix), 284 older. All live errors since 14:00 are covered by existing kwarg aliases in current code. Task acceptance criteria: batch triaged, fixes applied, counts documented.'str' object has no attribute 'get' error in paper_corpus_ingest (18 rows, Apr 10).PaperCorpus.ingest() at line 1347 calls paper.get("_provider") without checking if paper is a dict. Callers passing lists of strings cause TypeError.if not isinstance(paper, dict): continue guard in PaperCorpus.ingest() to skip non-dict items gracefully.corpus.ingest(['string', {'pmid': '123'}]) → {'ingested': 1, 'total': 2} ✓; corpus.ingest(['string1', 'string2']) → {'ingested': 0, 'total': 2} ✓.python3 -m py_compile scidex/forge/tools.py → ✓.304de4fe9. Remaining error patterns are dominated by stale pre-fix artifacts and upstream schema coverage gaps.origin/main (commit cde47a9a5) to eliminate stale local edits to api.py and api_routes/static_assets.py.msigdb_gene_sets(max_results), 12x pubchem_compound(search_type), 3x openalex_works(per_page), 3x gtex_tissue_expression(gene_symbol), 2x paper_corpus_search(query), plus 3x brainspan_expression(gene_symbol) and 3x expression_atlas_differential(gene_symbol).brainspan_expression(gene_symbol=None) — made gene_symbol kwarg with None default; added isinstance(gene_symbol, str) coercion before using in URL; added if not gene_symbol: return {...} guard.expression_atlas_differential(gene_symbol=None, max_results=20) — made gene_symbol kwarg with None default; added isinstance(max_results, int) coercion.openalex_works(query=None, per_page=None) — confirmed query already optional; confirmed per_page alias already wired; added isinstance(per_page, int) coercion.
python3 -m py_compile scidex/forge/tools.py → ✓; brainspan_expression() → {'gene': None, ...} ✓; expression_atlas_differential() → {'gene': None, ...} ✓; openalex_works() → {'query': None, ...} ✓; paper_corpus_search(max_results='5') → 5 total ✓; search_trials('cancer', status='RECRUITING') → 10 studies ✓.cc69f382c. Remaining untriaged count: 484 (dominated by stale pre-fix artifacts and upstream schema coverage gaps requiring skills.input_schema + skills.example_input coverage work).