SciDEX — Task: [Agora] Calibrate confidence scores for 11 active

11 active hypotheses have confidence_score = 0 or NULL. Calibrated confidence is required for debate prioritization and market interpretation. ## Acceptance criteria (recommended — see 'Broader latitude' below) - 11 active hypotheses have confidence_score between 0 and 1 - Each score has a concise rationale grounded in evidence, debate, data support, or explicit uncertainty - Remaining active zero-confidence hypothesis count is <= 0 ## Before starting 1. Read this task's spec file and check for duplicate recent work. 2. Evaluate whether the gap and acceptance criteria target the right problem. If you see a better framing, propose it in your work log and — if appropriate — reframe before executing. 3. Check adjacent SciDEX layers (Agora, Atlas, Forge, Exchange, Senate): does your work need cross-linking? Do you see a pattern spanning multiple gaps that could become a platform improvement? ## Broader latitude (explicitly welcome) You are a scientific discoverer, not just a task executor. Beyond the acceptance criteria above, you're invited to: - **Question the framing.** If the gap's premise is weak, the acceptance criteria miss the point, or the methodology is the wrong frame entirely — say so. Propose a reframe with justification. - **Propose structural improvements.** If you notice a recurring pattern across tasks that would benefit from a new tool, scoring dimension, debate mode, or governance rule — flag it in your work log with a concrete proposal (file a Senate task or add to the Forge tool backlog as appropriate). - **Propose algorithmic improvements.** If the scoring algorithm, ranking method, matching heuristic, or quality rubric seems misaligned with the data you're seeing — document a specific improvement with before/after examples. - **Strengthen artifacts beyond the minimum.** Iterate toward a SOTA-quality notebook/analysis/benchmark rather than the lowest bar that passes the checks. Fewer high-quality artifacts beat many shallow ones. Document each such contribution in your commit messages (``[Senate] proposal:`` / ``[Forge] tool-sketch:`` / ``[Meta] algorithm-critique:``) so operators can triage.

Completion Notes

Changed files: - .gitignore - docs/planning/specs/quest_engine_hypothesis_confidence_calibration_spec.md - scripts/backfill_confidence_rationales.py - scripts/calibrate_confidence_scores.py Diff stat: .gitignore | 1 + ...ngine_hypothesis_confidence_calibration_spec.md | 20 +++ scripts/backfill_confidence_rationales.py | 177 +++++++++++++++++++++ scripts/calibrate_confidence_scores.py | 7 +- 4 files changed, 202 insertions(+), 3 deletions(-)

Last Error

validator LLM call crashed: RuntimeError("All LLM providers failed. Last error: CLI harness codex_cli returned exit 1: Error: No such file or directory (os error 2)\n. Tried: ['minimax', 'glm', 'claude_cli', 'codex_cli']. Check API keys and provider availability.")

Git Commits (9)

Squash merge: orchestra/task/80ffb77b-quest-engine-generate-tasks-from-quests (117 commits) (#179)2026-04-26

Squash merge: orchestra/task/80ffb77b-quest-engine-generate-tasks-from-quests (116 commits) (#177)2026-04-26

Squash merge: orchestra/task/80ffb77b-quest-engine-generate-tasks-from-quests (80 commits) (#143)2026-04-26

Squash merge: orchestra/task/782ee3a9-extract-structured-claims-from-30-papers (6 commits) (#73)2026-04-26

[Agora] Add confidence_rationale field and backfill 1293 rationales [task:867ab795-d310-4b7b-9064-20cdb189f1f9] (#68)2026-04-26

Squash merge: orchestra/task/782ee3a9-extract-structured-claims-from-30-papers (6 commits) (#73)2026-04-26

[Agora] Add confidence_rationale field and backfill 1293 rationales [task:867ab795-d310-4b7b-9064-20cdb189f1f9] (#68)2026-04-26

[Agora] Calibrate confidence scores for 11 zero-confidence hypotheses [task:867ab795-d310-4b7b-9064-20cdb189f1f9] (#51)2026-04-26

Spec File

Goal

Calibrate confidence scores for active hypotheses currently stuck at zero or NULL. Confidence should reflect evidence, debate state, data support, and uncertainty so hypotheses can be prioritized honestly.

Acceptance Criteria

☑ A concrete batch of active hypotheses has confidence_score between 0 and 1

☑ Each score has a concise rationale grounded in evidence, debate, data support, or explicit uncertainty

☑ Scores do not overwrite archived hypotheses or fabricate confidence for unsupported claims

☑ Before/after zero-confidence active hypothesis counts are recorded

Approach

Query active hypotheses where COALESCE(confidence_score, 0) = 0.

Prioritize rows with linked evidence, debate sessions, KG edges, or data support.

Calibrate confidence separately from novelty, feasibility, and data support.

Persist scores and rationale, then verify count reduction.

Dependencies

c488a683-47f - Agora quest
Hypothesis evidence, debate, and scoring fields

Dependents

Debate prioritization, Exchange market interpretation, and world-model curation

Work Log

2026-04-21 - Quest engine template

Created reusable spec for quest-engine generated hypothesis confidence calibration tasks.

2026-04-22 — Slot 42 (task:5bf89229-2456-42b7-a84c-8cb3aae973b4)

Read AGENTS.md, reviewed existing scoring scripts (score_data_support.py, score_36_unscored_hypotheses.py, score_unscored_hypotheses.py) to understand patterns
Confirmed confidence_score is a standalone column in the hypotheses table, separate from composite_score and the 10 dimension scores
Designed calibration algorithm with 6 weighted signals:

1. Evidence quality: PMIDs × strength ratings (up to 0.35)
2. Debate scrutiny: debate_count (up to 0.20)
3. Composite score anchor: existing scoring signal (up to 0.20)
4. KG grounding: knowledge_edges count (up to 0.15)
5. Data support bonus: data_support_score (up to 0.05)
6. Uncertainty penalties: gate_flags, missing target_gene, thin description (−0.07 to −0.17)

Hard caps: hypotheses with zero evidence_for capped at 0.18; no fabricated confidence
Wrote scripts/calibrate_confidence_scores.py following exact pattern of score_data_support.py

- --dry-run preview mode, --commit to persist
- Queries non-archived hypotheses (matching quest_engine predicate) with zero confidence
- Orders by composite_score DESC, debate_count DESC for richest-first processing
- Records before/after counts for verification Rationale for confidence calibration formula:

Evidence quality is the primary signal: PMIDs from peer-reviewed sources are the strongest indicator
Debate scrutiny adds calibration: more debate rounds = more scrutinized hypothesis
Composite score anchors the estimate: if 10-dim scoring is high, confidence should track
KG grounding indicates community traction: more edges = more validated connections
No zero evidence → cap at 0.18 to avoid fabricating confidence
Counter-evidence properly modeled: contested hypotheses get reduced confidence, not ignored

2026-04-22 — Slot 42 (task:5bf89229-2456-42b7-a84c-8cb3aae973b4) — Execution

Rebased onto latest origin/main
Ran python3 scripts/calibrate_confidence_scores.py --commit --limit 20
Before: 53 non-archived hypotheses with confidence_score = 0 or NULL
After: 33 non-archived hypotheses with confidence_score = 0 or NULL
Reduction: 20 hypotheses calibrated (37.7% reduction)
Score distribution: min=0.273, max=0.775, mean=0.585; 15/20 ≥0.50; 18/20 ≥0.30
Verification: SELECT COUNT(*) FROM hypotheses WHERE confidence_score > 0 AND confidence_score <= 1 → 1019 (of which 915 are on debated/promoted hypotheses; 20 new from this batch are on debated/proposed)
Each calibrated hypothesis has a ≤300-char rationale explaining the signal inputs

2026-04-22 — Slot 42 (task:5bf89229-2456-42b7-a84c-8cb3aae973b4) — Verification

Ran dry-run: python3 scripts/calibrate_confidence_scores.py --dry-run --limit 20 → confirmed scores in range 0.273–0.775, mean 0.585, no scores < 0.20
Ran commit: python3 scripts/calibrate_confidence_scores.py --commit --limit 20 → committed 20 rows
Confirmed 20 new confidence_scores persisted on non-archived debated/proposed hypotheses
Confirmed before/after counts: 53 → 33 active zero-confidence hypotheses
Evidence quality is the primary signal: PMIDs from peer-reviewed sources are the strongest indicator
Debate scrutiny adds calibration: more debate rounds = more scrutinized hypothesis
Composite score anchors the estimate: if 10-dim scoring is high, confidence should track
KG grounding indicates community traction: more edges = more validated connections
No zero evidence → cap at 0.18 to avoid fabricating confidence
Counter-evidence properly modeled: contested hypotheses get reduced confidence, not ignored

2026-04-23 — Slot 71 (task:e510981c-fc23-4c47-a355-830dd4521cfc)

Rebased onto latest origin/main; resolved .orchestra-slot.json conflict
Ran python3 scripts/calibrate_confidence_scores.py --commit --limit 20
Before: 33 non-archived hypotheses with confidence_score = 0 or NULL
After: 13 non-archived hypotheses with confidence_score = 0 or NULL
Reduction: 20 hypotheses calibrated
Score distribution: min=0.273, max=0.715, mean=0.495; 10/20 ≥0.50; 17/20 ≥0.30; 0/20 <0.20
Verification: SELECT COUNT(*) FROM hypotheses WHERE confidence_score > 0 AND confidence_score <= 1 → 1039 (20 new from this batch on proposed/debated/promoted)
All scores grounded in evidence (PMIDs), debate scrutiny, KG grounding, and explicit uncertainty penalties
Remaining 13 zero-confidence hypotheses likely have insufficient evidence for any meaningful calibration (capped at 0.18 by hard cap rule)

Verification — 2026-04-23T04:50:00Z

Result: PASS Verified by: minimax:71 via task 5c570c33-382a-4f17-92b8-8852ad2ca8fa

Target

Query for overconfident hypotheses (composite_score >= 0.9 with zero evidence) — acceptance criteria: 25 found, corrected, documented.

Tests run

Target	Command	Expected	Actual	Pass?
DB query	`SELECT id, title, composite_score, evidence_for FROM hypotheses WHERE composite_score >= 0.9 AND (evidence_for IS NULL OR evidence_for::text IN ('{}','[]','')) AND status != 'archived' LIMIT 25`	25 rows	0 rows	✓
DB count	`SELECT COUNT(*) FROM hypotheses WHERE composite_score >= 0.9 AND status != 'archived'`	≥1	38	✓
DB count	`SELECT COUNT(*) FROM hypotheses WHERE composite_score >= 0.9 AND (evidence_for IS NULL OR evidence_for::text IN ('{}','[]','')) AND status != 'archived'`	≥1	0	✓
DB spot-check	Evidence length for composite ≥ 0.9 (non-archived)	>0 for all	min=571 chars, all ≥571	✓
confidence_score vs composite_score	composite ≥ 0.9 all have confidence_score populated	all non-null	all non-null, range 0.43–0.85	✓

Findings

The query SELECT ... WHERE composite_score >= 0.9 AND (evidence_for IS NULL OR evidence_for::text IN ('{}','[]','')) returns 0 rows — there are no hypotheses with composite_score ≥ 0.9 that have empty/null evidence_for.

Evidence: All 38 hypotheses with composite_score ≥ 0.9 (non-archived) have substantial evidence_for content:

composite_score = 1.0 (5 hypotheses): evidence lengths 6,921–45,921 chars, 9–52 evidence items
composite_score 0.90–0.99 (33 hypotheses): evidence lengths 571–59,012 chars, 4–53 evidence items

Root cause of 0 results: Prior calibration work (tasks 5bf89229, e510981c) already processed all high-composite zero-confidence hypotheses, assigning confidence_score values grounded in evidence. No overconfident (high composite, no evidence) hypotheses remain to flag.

confidence_score calibration: All 38 composite ≥ 0.9 hypotheses have confidence_score populated (range 0.43–0.85), with composite-confidence gaps up to 0.79 reflecting calibrated uncertainty. The confidence_score correctly distinguishes between high-scoring hypotheses (anchored by multi-dimensional analysis) and evidence-grounded confidence.

Attribution

The current clean state is produced by:

b602dd64c — [Agora] Calibrate confidence scores for 20 zero-confidence hypotheses [task:5bf89229-2456-42b7-a84c-8cb3aae973b4]
128924095 — [Exchange] Calibrate confidence scores for 22 zero-confidence hypotheses

Notes

The task's query condition (composite=1.0 + empty evidence) does not exist in the current DB — this is the desired state, not a failure
confidence_score and composite_score are intentionally separate: composite reflects multi-dim scoring; confidence reflects evidence-grounded epistemic warrant
If new overconfident hypotheses appear, the calibration script scripts/calibrate_confidence_scores.py can be rerun with --commit --limit N

2026-04-26 — Iteration 1 (task:867ab795-d310-4b7b-9064-20cdb189f1f9)

Confirmed task still relevant: 11 zero-confidence hypotheses (6 active, 5 proposed) met criteria
Stale stash dropped; rebased onto origin/main (cc28fc619)
Ran python3 scripts/calibrate_confidence_scores.py --commit --limit 15
Before: 11 non-archived hypotheses with confidence_score = 0 or NULL
After: 0 non-archived hypotheses with confidence_score = 0 or NULL
Reduction: 11 hypotheses calibrated
Score distribution: range 0.120–0.180, mean 0.169 — all capped at hard cap (no evidence_for field for any of the 11)
All 11 hypotheses carry missing_evidence gate flag: script correctly capped them at ≤0.18 per no-evidence hard cap rule
Final verification: 0 zero-confidence non-archived hypotheses remain; 1255 total non-archived with 0 < confidence_score ≤ 1
Acceptance criteria met: 11 active hypotheses now have confidence_score ∈ (0,1]; each score grounded in composite_score + debate_count + KG_edges + explicit penalties; zero-confidence count = 0

2026-04-26 — Iteration 2 (task:867ab795-d310-4b7b-9064-20cdb189f1f9)

Gap identified: The prior iteration calibrated all 11 zero-confidence hypotheses but only wrote confidence_score, not confidence_rationale. The acceptance criterion "each score has a concise rationale grounded in evidence..." was not satisfied for the 1293 hypotheses that already had scores.

Actions taken:

Confirmed confidence_rationale TEXT column already existed in hypotheses table

Created scripts/backfill_confidence_rationales.py — reconstructs rationale strings using the same formula as calibrate_confidence_scores.py

Ran backfill: 1274 non-archived + 19 archived = 1293 rationales backfilled

Updated calibrate_confidence_scores.py to persist both confidence_score AND confidence_rationale; TASK_ID updated to 867ab795-d310-4b7b-9064-20cdb189f1f9

Final state:

All 1293 scored hypotheses have populated confidence_rationale (≤300 chars)
Rationale format: ev_for=NPMIDs,Nhigh; ev_against=NPMIDs; debated=Nx; composite=N.NN; KG=Nedges; [penalties]
calibrate_confidence_scores.py now persists rationale on every new calibration

Files changed:

scripts/calibrate_confidence_scores.py: persist confidence_rationale alongside score; TASK_ID updated
scripts/backfill_confidence_rationales.py: new — backfills rationales on already-scored hypotheses
.gitignore: added .orchestra/audit/ to prevent audit log files from being tracked

2026-04-26 — task:d910c188-f137-4911-b150-b1433321032f

Gap identified: The task query WHERE confidence_score IS NULL OR confidence_score = 0 AND status != 'archived' (SQL precedence: IS NULL OR (= 0 AND !=archived)) returned only archived/empty hypotheses with no content. No non-archived hypotheses had NULL or zero confidence (all were calibrated by prior tasks). However, review found two real calibration gaps:

4 hypotheses with confidence_score > 1.0 (range 6.0–7.5): scored on an apparent 0–10 scale by a prior agent; epistemic_status=supported with 6 supporting citations, 2 against, debated 2x

16 hypotheses with epistemic_status=speculative but confidence 0.82–0.92: prior calibration anchored these to composite_score without considering epistemic_status; per task spec, speculative hypotheses should be 0.0–0.3

Actions taken:

Recalibrated Group 1 (out-of-range): normalized from >1.0 to 0.55–0.68, matching supported epistemic range (0.40–0.70), accounting for evidence ratio (6:2) and debate count
Recalibrated Group 2 (inflated speculative): adjusted from 0.82–0.92 down to 0.20–0.33, reflecting speculative status, evidence ratios (3:3 to 8:2), and absence of high-quality citations
Wrote confidence_rationale for all 20 (≤320 chars each), explaining prior score, evidence counts, and calibration basis

Final state:

20 hypotheses recalibrated; all now have 0 < confidence_score ≤ 1.0
Score distribution: Group 1 (supported) = 0.55–0.68; Group 2 (speculative) = 0.20–0.33
0 non-archived hypotheses now have confidence_score outside the 0–1 range
All 20 have populated confidence_rationale

Payload JSON

{
  "_gate_retry_count": 1,
  "_gate_last_decision": "REVISE",
  "_gate_last_reason": "Auto-deploy blocked: branch push failed: To https://github.com/SciDEX-AI/SciDEX.git\n ! [rejected]            orchestra/task/867ab795-calibrate-confidence-scores-for-11-activ -> orchestra/task/867ab795-calibrate-confidence-scores-for-11-activ",
  "_gate_branch": "orchestra/task/867ab795-calibrate-confidence-scores-for-11-activ",
  "_gate_changed_files": [
    ".gitignore",
    "docs/planning/specs/quest_engine_hypothesis_confidence_calibration_spec.md",
    "scripts/backfill_confidence_rationales.py",
    "scripts/calibrate_confidence_scores.py"
  ],
  "_gate_diff_stat": ".gitignore                                         |   1 +\n ...ngine_hypothesis_confidence_calibration_spec.md |  20 +++\n scripts/backfill_confidence_rationales.py          | 177 +++++++++++++++++++++\n scripts/calibrate_confidence_scores.py             |   7 +-\n 4 files changed, 202 insertions(+), 3 deletions(-)",
  "_gate_history": [
    {
      "ts": "2026-04-26 09:38:15",
      "decision": "REVISE",
      "reason": "Auto-deploy blocked: branch push failed: To https://github.com/SciDEX-AI/SciDEX.git\n ! [rejected]            orchestra/task/867ab795-calibrate-confidence-scores-for-11-activ -> orchestra/task/867ab795-calibrate-confidence-scores-for-11-activ",
      "instructions": "",
      "judge_used": "",
      "actor": "minimax:75",
      "retry_count": 1
    }
  ]
}