[Senate] Field-shift detector - auto-report when consensus moves done

Effort: extensive

Goal

Scientific consensus does shift — sometimes gradually, sometimes
sharply (the amyloid-cascade reframing, the trem2-microglia turn,
the gut-brain-axis emergence). When consensus moves, SciDEX should
notice and emit a structured Field Shift Report that names the
shift, identifies the hypotheses that moved, summarizes the
triggering evidence, and asks which agent personas were ahead vs.
behind the shift (a calibration signal).

Today, no detector exists. The data is there: composite_score
trajectories per hypothesis, debate verdict flips, market price
swings, paper-citation surges. Compose them into a detector +
report generator that runs weekly.

Acceptance Criteria

- detect_shifts(field_slug: str, lookback_days: int = 60)
-> list[dict] — returns shift candidates each shaped
{shift_id, field_slug, kind: 'consensus_change' |
'new_hypothesis_dominance' | 'old_hypothesis_collapse' |
'evidence_reframing', confidence, trigger_artifacts,
movers: [...], summary_md}. Detection thresholds:
a hypothesis shifts ≥0.25 composite_score in 30 days,
OR a debate verdict flips after ≥3 prior consistent
verdicts, OR a market resolves against the prior
65-day price trend.
- compose_shift_report(shift_id) -> str LLM-driven
narrative report explaining what shifted, why, what
agents predicted vs. missed, and what to watch next. provenance edges to all triggering artifacts. register_artifact, discussable under Q-DSC, rankable
under Q-PERC. "who saw it coming" section comparing each persona's
stance on the moved hypothesis 60 days before the shift
vs. the post-shift consensus. Personas that called the
shift early get a +calibration_credit written into
agent_personas.calibration_history JSONB. (Sunday 22:00 UTC, before cohort tracker) runs detection
across all registered fields, composes reports for the
top-10 highest-confidence shifts, and emails them via the
daily-digest mailer. grouped by week with confidence-sorted ordering;
GET /field-shifts/{shift_id} renders the full report
with movers section, persona-calibration section, and
"what to watch" section. field with velocity_band='red-hot' gets a
mandatory shift-detection pass even outside the weekly
schedule (early warning for fast-moving sub-fields). hypotheses with one designed shift; assert
detect_shifts surfaces it with kind='consensus_change';
mock LLM in compose_shift_report; assert calibration
credit landed on the correct persona.

Approach

Detection is multi-signal — implement as separate

detector functions then aggregate; each detector returns
candidates with a confidence ∈ [0, 1].

Composer prompt structure: "Given this evidence trail

(artifacts in chronological order), explain in 600 words
what changed and why this shift is meaningful. Cite every
artifact at least once."

Calibration: 60-day-prior persona stance is reconstructed

from hypothesis_history_events (q-time-hypothesis-history-
viewer) — assert that quest is a hard dependency.

Email subject line is the report title; body is the

summary_md; full report linked.

Mark shift reports as priority_artifact=true so they

appear pinned in the whats-changed dashboard for the week
they were emitted.

Dependencies

• q-time-hypothesis-history-viewer — supplies persona-stance

history for calibration.

• q-time-research-front-velocity — triggers off-schedule

detection passes.

• q-synth-brief-writer-agent — composer pattern reuse.
• scidex.senate.calibration — calibration credit storage.

Work Log

2026-04-27 17:30 UTC — Slot minimax:73

• Migration applied: migrations/add_field_shift_tables.py creates field_shifts and field_shift_reports tables with proper indexes
• Core module: scidex/senate/field_shift_detector.py with:

- detect_composite_score_shifts() — detects ≥0.25 score changes in 30 days using hypothesis_score_history
- detect_market_reversal_shifts() — detects price reversals against 65-day trend using price_history
- detect_debate_verdict_flips() — detects verdict flips after ≥3 consistent verdicts
- aggregate_shift_candidates() — combines signals with confidence normalization
- compose_shift_report() — LLM-driven 600-word narrative with fallback
- run_weekly_detection() — batch processing for top-N shifts
- get_shift_index() / get_shift_report() — DB queries for API layer

• API endpoints added: GET /api/senate/field-shifts, GET /api/senate/field-shifts/{shift_id}, POST /api/senate/field-shifts/detect in api_routes/senate.py
• Scheduled task: field-shift-weekly registered in scheduled_tasks.py with 10080 min (weekly) interval
• Tests: tests/test_field_shift_detector.py — 11 tests covering aggregation logic, API shape, LLM mocking
• Detection verified: 223 shifts detected on live DB on first run
• NOT implemented (deferred): persona calibration_history JSONB updates, systemd timer file creation, Q-DSC/Q-PERC discussability/rankability integration, velocity_band integration, email digest