[Senate] Capture belief snapshots for hypotheses missing recent state

Goal

Backfill recent belief snapshots for hypotheses that lack current time-series state. Belief snapshots let SciDEX measure convergence, evidence accumulation, and confidence changes over time.

Acceptance Criteria

Approach

Dependencies

• q-epistemic-rigor - Epistemic Rigor quest

Dependents

• Epistemic rigor metrics, convergence dashboards, and Senate retrospectives

Work Log

2026-04-24 14:50 UTC — task:0c46779c

• Reviewed the live PostgreSQL schema via scidex.core.database.get_db() before changing anything.
• Confirmed belief_snapshots currently stores score, evidence_count, citations, and reviewer_count; the generated task wording mentioning market_price and confidence_score is ahead of the live schema.
• Verified the real hypothesis status set is proposed / promoted / debated / open / archived; there is no active status in the current data.
• Measured the live backlog using non-archived hypotheses and a 7-day window: 548 hypotheses were missing recent snapshots before this run.
• Planned implementation for this run: update scripts/backfill_belief_snapshots.py to use the 7-day window, target non-archived hypotheses, default to a batch size of 30, and keep same-day inserts idempotent.

• Executed python3 scripts/backfill_belief_snapshots.py --limit 30.
• Before/after backlog (7-day window, excluding archived / rejected / resolved): 548 -> 518.
• Verified 30 belief_snapshots rows were written on 2026-04-24 and 0 same-day duplicate hypothesis snapshots exist.
• Spot-checked inserted rows against live hypotheses state: stored score matched composite_score, stored evidence_count matched live evidence array counts, and the sampled rows’ live market_price / confidence_score values were recorded during execution for audit context.

2026-04-21 - Quest engine template

• Created reusable spec for quest-engine generated belief snapshot backfill tasks.

2026-04-22 — task:4cfdd27b — Backfill 30 hypotheses

• Before count: 679 hypotheses missing belief snapshots (30-day window)
• Inserted 30 new snapshots via direct SQL INSERT with NOT EXISTS guard
• Mapping: score ← composite_score, evidence_count ← len(evidence_for)+len(evidence_against), citations ← citations_count, reviewer_count ← 0
• After count (1-day window): 300 snapshots (up from 270 pre-run)
• Remaining missing (30-day): 649

INSERT INTO belief_snapshots (hypothesis_id, score, evidence_count, citations, reviewer_count, snapshot_date)
SELECT h.id,
  COALESCE(h.composite_score, 0.5),
  COALESCE(jsonb_array_length(COALESCE(h.evidence_for, '[]'::jsonb)), 0) +
  COALESCE(jsonb_array_length(COALESCE(h.evidence_against, '[]'::jsonb)), 0),
  COALESCE(h.citations_count, 0),
  0,
  NOW()
FROM hypotheses h
WHERE NOT EXISTS (
  SELECT 1 FROM belief_snapshots bs
  WHERE bs.hypothesis_id = h.id AND bs.snapshot_date > NOW() - INTERVAL '30 days'
)
ORDER BY h.created_at DESC LIMIT 30;
-- Result: INSERT 0 30

2026-04-21 14:00-14:05 UTC — Backfill execution

• Before count: 830 hypotheses missing belief snapshots (7-day window)
• After count: 730 hypotheses missing belief snapshots (7-day window)
• Reduction: 100 (exceeds 50 target due to concurrent quest_engine runs)
• Created scripts/backfill_belief_snapshots.py with idempotent INSERT using WHERE NOT EXISTS clause
• 50 specific hypothesis IDs received belief_snapshots rows via the script
• Snapshots include: score (composite_score), evidence_count (len(evidence_for) + len(evidence_against)), citations (citations_count), reviewer_count (hypothesis_reviews count)
• Same-day duplicates avoided via conditional INSERT
• Remaining missing (730) is well under the 780 threshold

-- Before/after counts
SELECT COUNT(*) FROM hypotheses h WHERE NOT EXISTS (SELECT 1 FROM belief_snapshots b WHERE b.hypothesis_id = h.id AND b.snapshot_date >= CURRENT_DATE - INTERVAL '7 days');

-- Verify 50 new snapshots with all required fields
SELECT bs.hypothesis_id, bs.score, bs.evidence_count, bs.citations, bs.reviewer_count, bs.snapshot_date
FROM belief_snapshots bs WHERE bs.snapshot_date::date = CURRENT_DATE ORDER BY bs.id DESC LIMIT 50;

-- Verify no duplicates today
SELECT hypothesis_id, COUNT(*) FROM belief_snapshots WHERE snapshot_date::date = CURRENT_DATE GROUP BY hypothesis_id HAVING COUNT(*) > 1;