SciDEX — Task: [Senate] Unblock 10 stuck debates in the mission p

176 debates have aged past their SLA window with no downstream output. Pipeline stalling blocks world-model growth. ## Acceptance criteria (recommended — see 'Broader latitude' below) - 10 stuck debates are diagnosed and advanced or closed with rationale - Root cause is identified (missing data, missing debate, missing linkage, etc.) - Remaining stuck debates count is <= 166 ## Before starting 1. Read this task's spec file and check for duplicate recent work. 2. Evaluate whether the gap and acceptance criteria target the right problem. If you see a better framing, propose it in your work log and — if appropriate — reframe before executing. 3. Check adjacent SciDEX layers (Agora, Atlas, Forge, Exchange, Senate): does your work need cross-linking? Do you see a pattern spanning multiple gaps that could become a platform improvement? ## Broader latitude (explicitly welcome) You are a scientific discoverer, not just a task executor. Beyond the acceptance criteria above, you're invited to: - **Question the framing.** If the gap's premise is weak, the acceptance criteria miss the point, or the methodology is the wrong frame entirely — say so. Propose a reframe with justification. - **Propose structural improvements.** If you notice a recurring pattern across tasks that would benefit from a new tool, scoring dimension, debate mode, or governance rule — flag it in your work log with a concrete proposal (file a Senate task or add to the Forge tool backlog as appropriate). - **Propose algorithmic improvements.** If the scoring algorithm, ranking method, matching heuristic, or quality rubric seems misaligned with the data you're seeing — document a specific improvement with before/after examples. - **Strengthen artifacts beyond the minimum.** Iterate toward a SOTA-quality notebook/analysis/benchmark rather than the lowest bar that passes the checks. Fewer high-quality artifacts beat many shallow ones. Document each such contribution in your commit messages (``[Senate] proposal:`` / ``[Forge] tool-sketch:`` / ``[Meta] algorithm-critique:``) so operators can triage.

Completion Notes

Auto-completed by supervisor after successful deploy to main

Git Commits (2)

[Senate] Unblock stuck mission-pipeline debates [task:033e0b91-cd31-44dc-b3c4-58a1a1e1912b] (#56)2026-04-26

[Senate] Unblock stuck mission-pipeline debates [task:033e0b91-cd31-44dc-b3c4-58a1a1e1912b]2026-04-26

Spec File

Goal

Extend quest_engine.py:discover_gaps() with five mission-pipeline gap detectors that balance the queue toward feature flow, not just data backfill. Today the engine only reseeds data-backfill gaps; these detectors let the engine balance its own outputs toward flow by detecting stuck stages, throughput imbalances, and missing connector tasks.

Acceptance Criteria

☐ stuck_stage_gap: any pipeline stage where N >= 10 items are aged-out → spawns 'unblock <stage>' task at pri 88

☐ low_landscape_to_gap_throughput: if landscapes added in last 14d > gaps spawned * 2 → seed 'extract gaps from N recent landscapes' at pri 87

☐ low_gap_to_debate_throughput: if quality-scored gaps > debates_run * 3 → seed 'fan out debates for N high-q gaps' at pri 87

☐ low_hypothesis_to_action_throughput: if top-ranked hypotheses (composite_score >= 7) > (challenges + experiment_proposals) → seed 'create N challenges or proposals' at pri 87

☐ missing_bridge: if a connector spec exists in docs/planning/specs/mission_*.md but no corresponding task is open or done → seed task referencing spec

☐ Each detector follows the existing Gap dataclass shape and lives in discover_gaps()

☐ Unit tests in tests/quest_engine/test_mission_pipeline_gaps.py

Approach

1. Study the existing pattern

Existing gap detectors in quest_engine.py:discover_gaps() follow this pattern:

Call scalar(conn, sql) to get a count
If count > 0, append a Gap(...) object
Gap fields: key, layer, priority, count, title, summary, acceptance, approach, requires

2. Add mission-pipeline gap detectors

Add five new gap detectors before the return sorted(gaps, ...) line at line ~1546:

stuck_stage_gap

Query: counts aged-out items per pipeline stage
Stage definitions from mission_pipeline_observability.md:

- Landscapes: no linked gaps after creation, > 7d old
- Gaps: status='open', last_debated_at IS NULL, > 30d old
- Debates: num_hypotheses_generated=0, completed_at IS NOT NULL, > 3d since completion
- Hypotheses: composite_score < 0.5, > 14d old
- Ranked: composite_score >= 0.5, no challenge linked, > 30d old
- Challenges/Experiments: status='open', no linked experiments, > 45d old

Condition: N >= 10 → spawn 'unblock <stage>' at pri 88
Layer: Senate (pipeline health)

low_landscape_to_gap_throughput

Count landscapes added in last 14d
Count gaps spawned (knowledge_gaps with created_at in last 14d)
Condition: landscapes_14d > gaps_spawned_14d * 2 → seed at pri 87
Layer: Atlas

low_gap_to_debate_throughput

Count quality-scored gaps (gap_quality_score IS NOT NULL, status NOT IN closed/archived)
Count debates run in last 30d (debate_sessions with completed_at in last 30d)
Condition: quality_gaps > debates_run_30d * 3 → seed at pri 87
Layer: Agora

low_hypothesis_to_action_throughput

Count top-ranked hypotheses: composite_score >= 0.7, status NOT archived
Count challenges + experiment_proposals linked to those hypotheses
Condition: top_hypotheses > linked_challenges + linked_experiments → seed at pri 87
Layer: Exchange

missing_bridge

Scan docs/planning/specs/mission_*.md files for connector specs
Check orchestra DB for open/done tasks referencing each spec path
Condition: spec exists with connector intent but no open/done task → seed at pri 85
Layer: Senate

3. Unit tests

Create tests/quest_engine/test_mission_pipeline_gaps.py:

Mock get_scidex_db() with fixture DB
Test each detector function with controlled row counts
Test threshold conditions (above/below)
Test that Gap objects have correct priority, layer, key

Dependencies

quest_engine.py — discover_gaps(), Gap dataclass, scalar()
scidex.core.database — get_db_readonly()
Existing mission specs in docs/planning/specs/mission_*.md
mission_pipeline_observability.md — stage/stuck definitions

Work Log

2026-04-26 — Implementation

Created spec file
Added five gap detectors to discover_gaps()
Added SPEC_PATHS entries for new gap keys
Created unit tests in tests/quest_engine/test_mission_pipeline_gaps.py

2026-04-26 — Task 033e0b91 stuck-debate remediation

Ran python3 scripts/unblock_stuck_gap_debates.py --limit 10: reconciled 2 completed debates with existing hypotheses, recovered 4 archived analyses back to completed with 16 total hypotheses, closed 3 irrecoverable debates with explicit failure reasons, and confirmed 1 older archived parse-error case was already closed.
Live DB verification after remediation: the legacy detector dropped from 176 to 170 rows, the real gap_analysis backlog dropped from 17 to 11, and the corrected detector query returned 0 active stuck mission-pipeline debates.
Reframed the debate portion of the mission-pipeline detector after validating live PostgreSQL state: 176 flagged rows split into 159 target_debate sessions with no analysis_id and 17 real gap_analysis sessions.
Tightened the detector semantics so only mission-pipeline gap_analysis sessions with no linked hypotheses and non-archived analyses count as stuck debates.
Added scripts/unblock_stuck_gap_debates.py to reconcile stale debate counters, recover analyses with valid synthesizer payloads, and carry forward explicit closure rationale for irrecoverable sessions.