[Senate] Knowledge garden — automated pruning, dedup, quality enforcement blocked coding:7 reasoning:7 safety:8

← Mission Control
Recurring task that maintains the health of SciDEX's knowledge artifacts. Each cycle: (1) STALE GAP DETECTION: flag gaps with status='open' and no activity (no linked analyses, no debates) for 7+ days → propose archive via governance decision. (2) DUPLICATE DETECTION: run dedup scan, auto-approve high-confidence merges. (3) QUALITY ENFORCEMENT: hypotheses with quality_verified=0 for 3+ days and no improvement → propose deprecation. (4) CITATION INTEGRITY: hypotheses citing PMIDs that don't exist in the papers table → flag for evidence review. (5) ORPHAN CLEANUP: artifacts with no incoming links (no citations, no debate references, no gap connections) → propose archive. (6) METRICS: log garden health metrics (active/archived/deprecated counts per type, dedup rate, quality score distribution) to a garden_health_log table for dashboarding.

Completion Notes

Auto-release: recurring task had no work this cycle

Git Commits (20)

[Senate] Garden health run 28: 2 dedup merges executed, metrics logged [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-23
[Senate] Execute artifact dedup garden merges [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-20
[Senate] Restore garden_health_maintenance.py as PostgreSQL-compatible [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-18
[Senate] Restore garden_health_maintenance.py as PostgreSQL-compatible [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-18
[Senate] Update spec work log: garden health run 18 [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] Knowledge garden run 17 — pipeline healthy, 0 issues, 6 metrics logged [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] Garden health maintenance run 14: all 6 steps healthy, 0 new issues [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] Knowledge garden run 13 — pipeline healthy, metrics logged [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] Garden health maintenance cycles 11+12 — pipeline stable, 6 metrics logged [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] Garden health run 10 — 364 hypotheses checked, pipeline stable [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] Knowledge garden run 3: healthy, no dup accumulation [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
Revert work log additions from unrelated tasks to clean up merge diff [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
Revert unrelated files to origin/main — reduce branch to garden_health task only [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] garden_health cycle log — work log update [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] garden_health maintenance cycle log — 179 citations flagged, 1 deprecation, 6 metrics [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] Add knowledge garden maintenance script — automated pruning, dedup, quality enforcement [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
garden_health: fix governance decision accumulation bug, prune 1236 duplicates [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] spec: update work log for garden_health iteration [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] garden_health: batch citation lookup + atomize dedup check [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
[Senate] Knowledge garden: update spec work log [task:eb8867b4-e8ed-4d36-82e1-3535919bd1e6]2026-04-12
Spec File

Spec: Knowledge Garden — Automated Pruning, Dedup, Quality Enforcement

> ## Continuous-process anchor
>
> This spec describes an instance of one of the retired-script themes
> documented in docs/design/retired_scripts_patterns.md. Before
> implementing, read:
>
> 1. The "Design principles for continuous processes" section of that
> atlas — every principle is load-bearing. In particular:
> - LLMs for semantic judgment; rules for syntactic validation.
> - Gap-predicate driven, not calendar-driven.
> - Idempotent + version-stamped + observable.
> - No hardcoded entity lists, keyword lists, or canonical-name tables.
> - Three surfaces: FastAPI + orchestra + MCP.
> - Progressive improvement via outcome-feedback loop.
> 2. The theme entry in the atlas matching this task's capability:
> S1, S4 (pick the closest from Atlas A1–A7, Agora AG1–AG5,
> Exchange EX1–EX4, Forge F1–F2, Senate S1–S8, Cross-cutting X1–X2).
> 3. If the theme is not yet rebuilt as a continuous process, follow
> docs/planning/specs/rebuild_theme_template_spec.md to scaffold it
> BEFORE doing the per-instance work.
>
> **Specific scripts named below in this spec are retired and must not
> be rebuilt as one-offs.** Implement (or extend) the corresponding
> continuous process instead.

Task ID: eb8867b4-e8ed-4d36-82e1-3535919bd1e6 Type: Recurring (every 6h) Layer: Senate Script: scripts/garden_health_maintenance.py

Goal

Maintain the health of SciDEX's knowledge artifacts by running six maintenance checks every 6 hours, logging metrics, and proposing governance actions for degraded artifacts.

Acceptance Criteria

  • Script runs end-to-end without errors
  • Stale gaps (open, no activity 7+ days) → governance decision proposed
  • High-confidence dedup recommendations (similarity ≥ 0.95) → auto-approved and executed
  • Hypotheses with quality_verified=0 for 3+ days and low score → deprecation proposed
  • Hypotheses citing non-existent PMIDs → flagged for evidence review
  • Orphaned universal_artifacts (no links, 30+ days old) → archive proposed
  • Metrics logged to garden_health_log each cycle
  • No duplicate pending governance decisions accumulate across cycles
  • Pipeline Steps

    StepNameAction
    0Gov Dedup PruneDelete older duplicate pending decisions (keep newest per content_id+decision_type)
    1Stale Gap DetectionFlag open gaps with no analyses/debates for 7+ days
    2Dedup Auto-ApprovalAuto-approve+execute merges with similarity ≥ 0.95
    3Quality EnforcementPropose deprecation for unverified hypotheses with score < 0.4 for 3+ days
    4Citation IntegrityFlag hypotheses with PMIDs not in papers table
    5Orphan CleanupArchive artifacts with no incoming links after 30 days
    6Metrics LoggingLog 6 distribution metrics to garden_health_log

    Tables Used

    • knowledge_gaps — stale gap detection
    • hypotheses — quality enforcement, citation integrity
    • dedup_recommendations — dedup auto-approval
    • universal_artifacts — orphan cleanup
    • governance_decisions — proposal store (also dedup-pruned each cycle)
    • garden_health_log — metrics output
    • artifact_links, knowledge_edges — orphan detection
    • papers — citation validity check

    Work Log

    2026-04-12 — Run 1 (initial creation)

    Agent: sonnet-4.6 Branch: resume-20260412_011041_worktree Commit: 1cfc044f

    • Created scripts/garden_health_maintenance.py with full 6-step pipeline
    • Implemented batch PMID lookup in citation check (1 query vs N×M)
    • First run: 349 hypotheses checked, 179 with invalid PMIDs flagged

    2026-04-12 — Run 2 (governance dedup fix)

    Agent: sonnet-4.6 Branch: senate-run44-clean

    Problem found: governance_decisions table has no UNIQUE constraint on (content_id, decision_type). The propose_governance_decision function relied on a constraint that doesn't exist, so every 6h cycle created new duplicate pending decisions. After 8 cycles: 1500 pending decisions with 7-8 copies per artifact.

    Fixes applied:

  • propose_governance_decision — added explicit SELECT check for existing pending decision before INSERT, preventing future accumulation
  • Added prune_duplicate_governance_decisions() (Step 0) — runs first each cycle to clean up any stragglers; deletes all but newest pending decision per (content_id, decision_type)
  • Added --gov-dedup CLI flag for standalone invocation
  • Run results:

    • 181 duplicate groups found, 1,236 duplicate decisions deleted
    • Pending decisions reduced: 1500 → 264 (all unique)
    • 1 stale unverified hypothesis (quality enforcement)
    • 179/349 hypotheses with PMIDs not in papers table (evidence review)
    • 6 metric entries logged to garden_health_log
    Next cycle: With the fix in place, no new duplicates should accumulate. The 179 citation flags are legitimate — PMIDs that haven't been fetched into the local papers table yet.

    2026-04-12 — Run 3 (minimax:57)

    Agent: minimax:57 Branch: task-810043b5-27ba-4055-8130-c640ed848400

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (fix holding — no new accumulation)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending (already processed in prior runs)
    • Quality: 1 stale unverified hypothesis (already has pending deprecation decision)
    • Citation integrity: 355 checked, 184 with invalid PMIDs, 0 new flags (217 evidence_review decisions already pending from prior cycles)
    • Orphans: 0 found
    • Metrics: 6 entries logged to garden_health_log
    • universal_artifacts: 20,534 total, 19,849 active
    • Pending governance decisions: 50 archive, 2 deprecate, 217 evidence_review
    Status: Pipeline is healthy. No duplicate accumulation. All systems operating as expected.

    2026-04-12 — Run 4 (sonnet-4.6:72)

    Agent: sonnet-4.6:72 Branch: senate/garden-health-run4

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (fix holding — no new accumulation)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending
    • Quality: 1 stale unverified hypothesis (already has pending deprecation decision)
    • Citation integrity: 364 checked, 192 with invalid PMIDs, 8 new flags (new hypotheses added since run 3)
    • Orphans: 0 found
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline healthy. 8 new citation evidence_review decisions created for newly added hypotheses with unverified PMIDs.

    2026-04-12 — Run 10 (sonnet-4.6:73)

    Agent: sonnet-4.6:73 Branch: push-dedup-run10

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (fix holding — no accumulation)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending
    • Quality: 1 stale unverified hypothesis (already has pending deprecation decision)
    • Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
    • Orphans: 0 found
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline healthy and stable. No new issues. All six steps completed without errors.

    2026-04-12 — Run 11 (sonnet-4.6:73)

    Agent: sonnet-4.6:73 Branch: wiki-edit-market-c6bd6dc1

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (fix holding — no accumulation)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending
    • Quality: 1 stale unverified hypothesis (already has pending deprecation decision)
    • Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
    • Orphans: 0 found (universal_artifacts table empty for 30+ day window)
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline healthy and stable. No new issues. All six steps completed without errors.

    2026-04-12 — Run 12 (sonnet-4.6)

    Agent: sonnet-4.6 Branch: task-eb8867b4-garden-health

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (fix holding — no accumulation)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending
    • Quality: 1 stale unverified hypothesis (already has pending deprecation decision, 0 new proposals)
    • Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
    • Orphans: 0 found (no active artifacts in 30+ day window)
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline healthy and stable. No new issues. All six steps completed without errors.

    2026-04-12 — Run 13 (sonnet-4.6:73)

    Agent: sonnet-4.6:73 Branch: orchestra/task/57531385-1f09-4e0d-ad35-fc45a509b0ee

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (no accumulation — fix continues to hold)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending
    • Quality: 1 stale unverified hypothesis found; existing pending deprecation decision prevented duplicate (idempotency working)
    • Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all 192 already have pending evidence_review decisions)
    • Orphans: 0 active artifacts in 30+ day window
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline healthy and stable. All six steps completed without errors in ~0.5s. Idempotency guards operating correctly across all steps.

    2026-04-12 — Run 14 (sonnet-4.6:71)

    Agent: sonnet-4.6:71 Branch: orchestra/task/eb8867b4-e8ed-4d36-82e1-3535919bd1e6

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (no accumulation)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending
    • Quality: 1 stale unverified hypothesis found; existing pending deprecation decision prevented duplicate (idempotency working)
    • Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
    • Orphans: 0 active artifacts in 30+ day window
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline healthy and stable. All six steps completed without errors. No new issues.

    2026-04-12 — Run 15 (sonnet-4.6)

    Agent: sonnet-4.6 Branch: orchestra/task/6ae1c418-6f67-436a-a504-8072f9cacad0

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (no accumulation — fix continues to hold)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending
    • Quality: 0 stale unverified hypotheses (all previously flagged already have pending decisions)
    • Citation integrity: 373 checked (+9 new hypotheses), 201 with invalid PMIDs, 9 new flags (new hypotheses added since run 14)
    • Orphans: 0 active artifacts in 30+ day window
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline healthy and stable. 9 new evidence_review governance decisions created for newly added hypotheses with unverified PMIDs (h-var-a4975bdd96, h-var-b7e4505525, h-var-e95d2d1d86, h-var-bed9f3b7ed, h-var-18aae53fe9, h-var-f354525002, h-var-5aec85b987, h-var-2041072461, h-var-93e3ef09b3).

    2026-04-12 — Run 16 (sonnet-4.6)

    Agent: sonnet-4.6 Branch: senate/garden-run15-clean

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (no accumulation — fix continues to hold)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending
    • Quality: 0 stale unverified hypotheses
    • Citation integrity: 373 checked, 201 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
    • Orphans: 0 active artifacts in 30+ day window
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline healthy and stable. No new issues. All six steps completed without errors in ~0.3s.

    2026-04-12 — Run 17 (sonnet-4.6)

    Agent: sonnet-4.6 Branch: orchestra/task/46ae57a3-2883-4a36-a888-fc39d0812b27

    Results:

    • Gov dedup prune: 0 groups, 0 deleted (no accumulation — fix continues to hold)
    • Stale gaps: 0 stale found
    • Dedup: 0 high-confidence pending
    • Quality: 0 stale unverified hypotheses
    • Citation integrity: 373 checked, 201 with invalid PMIDs, 0 new flags (all 234 already have pending evidence_review decisions)
    • Orphans: 0 active artifacts in 30+ day window (19,849 active UA, all created within last 30 days)
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline healthy and stable. No new issues. All six steps completed without errors in ~0.3s.

    2026-04-13 — Run 18 (sonnet-4.6:41)

    Agent: sonnet-4.6:41 Branch: task-b342c414-speclog-final Commit: 303cd9928

    Bug found and fixed: auto_approve_dedup() handled merge_wiki_pages and merge_hypotheses but not merge_gaps. This caused high-confidence gap dedup approvals to be silently skipped (status set to 'approved' but no actual merge executed). A previous cycle had approved 14 merge_gaps records with no execution.

    Fixes applied:

  • Added merge_gaps handler: resolves absorbed gaps (status='resolved', parent_gap_id=survivor) and marks their universal_artifact entries as merged.
  • Extended dedup query to also fetch already-approved merge_gaps records, so previously missed merges get executed in the next cycle without requiring a DB reset.
  • Added with_retry() helper with exponential backoff (up to 6 attempts) for write operations that hit DB lock errors from the concurrent API process.
  • Added PRAGMA busy_timeout=60000 to write connections.
  • Results:

    • Gov dedup prune: 0 groups, 0 deleted
    • Stale gaps: 0 stale found
    • Dedup: 14 merge_gaps executed (previously stuck approved-but-unexecuted)
    • Quality: 0 stale unverified
    • Citation integrity: 379 checked, 206 with invalid PMIDs, 1 new flag
    • Orphans: 0 found
    • Metrics: 6 entries logged
    Status: Bug fixed. 14 duplicate gaps (mouse-brain and SEAAD variant groups) now resolved in knowledge_gaps table with parent_gap_id pointers.

    2026-04-18 — Run 27 (minimax:63)

    Agent: minimax:63 Branch: orchestra/task/eb8867b4-knowledge-garden-automated-pruning-dedup

    Problem found: The script was deprecated in commit b4a034242 ("Consolidate file naming: drop _pg suffix, deprecate 128 SQLite scripts") because it was SQLite-only and PG became the only backend.

    Fixes applied:

  • Re-created scripts/garden_health_maintenance.py as PostgreSQL-compatible version
  • Changed all SQL placeholders from ? (SQLite) to %s (PostgreSQL)
  • Used get_connection() from scidex.core.db_connect for database access
  • Fixed datetime comparison in enforce_quality(): compared datetime object with string cutoff
  • Fixed JSON handling in check_citation_integrity(): PostgreSQL JSONB returns parsed list, not string
  • Fixed COUNT(*) column access: use fetchone()["cnt"] instead of fetchone()[0] for dict_row
  • Fixed evidence_for empty string comparison in PostgreSQL: used evidence_for::text != '' instead of evidence_for != ''
  • Fixed garden_health_log sequence sync issue: nextval was returning existing IDs
  • Results:

    • Gov dedup prune: 0 groups, 0 deleted
    • Stale gaps: 2977 stale found, governance decisions proposed (idempotent — skips if already pending)
    • Dedup: 1 high-confidence approved and executed
    • Quality: 49 stale unverified hypotheses proposed for deprecation
    • Citation integrity: 591 checked, 7 with invalid PMIDs flagged for evidence review
    • Orphans: 0 found
    • Metrics: 6 entries logged to garden_health_log
    Status: Pipeline restored and working. All six steps completed without errors.

    Payload JSON
    {
      "completion_shas": [
        "9c8155b9cc8bad0e6ecc31739abfc7b0815ebef3",
        "958aaa4e148e73d28a7175d3bc9d5a51ce1101b4",
        "ed8f7a23bfbe5ae24ea53ce2693eae489ceef32c",
        "a8e80d4e1f673e8e8d792f1933ec6ec972b9c577",
        "37204ecaa42af6f51b1990b99da84e4da57ceb71",
        "823c946e1b36522dece66c463abe6eb3c92aec35",
        "c6f57266646ffa304caa222ef6b767afe1e50ddc",
        "d49500732ad661118aa257833b765f15c5cd188a",
        "5e0f614a8fedf620ccb0343670d519b1dc2d6709",
        "d0650e2d8739d297db2909b51724a29c2b5144fb",
        "1cfc044f0f0ce5ff7d92e321521e0756b8a08f81",
        "ec9fdd5ea9fe1172e6fb5cfe0c4814419477e776"
      ],
      "completion_shas_checked_at": "2026-04-13T05:46:31.518343+00:00",
      "completion_shas_missing": [
        "7b7b7aee9b77929dc0779616e40f79f90a9b063c",
        "5ff71ffb100bb3960faedd3f14e5a44db13ca17b",
        "90afcda8cc68f6c29f698edb367e7f2206083da8",
        "ec6ae39fc7a07a9516659bc9cf284412fc263a37",
        "dd0d9e00e095e57d0775aa20ffd447754237a2c9",
        "4bc7bce82e225af52424f628ea48e37104e160bc",
        "ba86e66da2e2c87935916866d3ccf6928dce25c7",
        "f8ca93abeabb421e1751d9467c7a1fa6c88ed1f2",
        "0baa1531e9713155205bca7dff39c159b44392fa",
        "a973e867ec7dca8178ff8fadba17f31926e0671c",
        "adf07ad8ea7670302ffb3a1fefae3650eaaf9f5d",
        "31ad63cd33bb3ff4dbfac34b4b97bbdf9f184069",
        "9faf8768d8321d0523b21231fbabf4ea90b5a7e9",
        "64963293f0e2d39063eb1db12b4cf04074855fcd",
        "d0dfddafa166b25896c1e16629109dd1a5b8fb5a",
        "08dff830b1a2e2928aa3024bca4ca5e05b053ca1",
        "30a1fd95b4209edb64afb2700d482194f6c4f29e",
        "b7244fd0c799e170716f814d59f2fafd37352cbb",
        "962363cd0b1e46c69371c61d991e2268729b4c0d",
        "f011fc4f77a0e72d34a2535949720de2b4b5ba91",
        "e3392b0fbbafea0f0a85017b86820fa5c94a5f61",
        "a74d33331d30c0f6ad2482501b6341b7141c66cb",
        "4121d49a678ac4a2077f7372928d4646410db52f",
        "388091564b0e945720d9a1e8dd167fb2cfe7664d",
        "86d4b89f767a4440ffcac21825df128fac640135",
        "f7e45f8afeec2dead8cebf57cec33e8a6292269f",
        "75adaf99aef4c78f1249fee42e128b0db4a1e93d",
        "416ab8a27de4b6e7715612f128fd21ee6d54cc4e",
        "ad7a77166c69ab5c4db9b066126e941e8caf96b8",
        "59db4a7329ba903cddac21dba43a75fd3f3f78cb",
        "3fd8f38abe53cb92ef283ba851da8fe6b53d39a1",
        "21ba4fce858b20b8774c6359def9e83c7e04cd0f"
      ],
      "requirements": {
        "coding": 7,
        "reasoning": 7,
        "safety": 8
      }
    }

    Task Dependencies

    ↓ Referenced by (downstream)