Spec: Knowledge Garden — Automated Pruning, Dedup, Quality Enforcement

> ## Continuous-process anchor
>
> This spec describes an instance of one of the retired-script themes
> documented in docs/design/retired_scripts_patterns.md. Before
> implementing, read:
>
> 1. The "Design principles for continuous processes" section of that
> atlas — every principle is load-bearing. In particular:
> - LLMs for semantic judgment; rules for syntactic validation.
> - Gap-predicate driven, not calendar-driven.
> - Idempotent + version-stamped + observable.
> - No hardcoded entity lists, keyword lists, or canonical-name tables.
> - Three surfaces: FastAPI + orchestra + MCP.
> - Progressive improvement via outcome-feedback loop.
> 2. The theme entry in the atlas matching this task's capability:
> S1, S4 (pick the closest from Atlas A1–A7, Agora AG1–AG5,
> Exchange EX1–EX4, Forge F1–F2, Senate S1–S8, Cross-cutting X1–X2).
> 3. If the theme is not yet rebuilt as a continuous process, follow
> docs/planning/specs/rebuild_theme_template_spec.md to scaffold it
> BEFORE doing the per-instance work.
>
> **Specific scripts named below in this spec are retired and must not
> be rebuilt as one-offs.** Implement (or extend) the corresponding
> continuous process instead.

Task ID: eb8867b4-e8ed-4d36-82e1-3535919bd1e6 Type: Recurring (every 6h) Layer: Senate Script: scripts/garden_health_maintenance.py

Goal

Maintain the health of SciDEX's knowledge artifacts by running six maintenance checks every 6 hours, logging metrics, and proposing governance actions for degraded artifacts.

Acceptance Criteria

Script runs end-to-end without errors

Stale gaps (open, no activity 7+ days) → governance decision proposed

High-confidence dedup recommendations (similarity ≥ 0.95) → auto-approved and executed

Hypotheses with quality_verified=0 for 3+ days and low score → deprecation proposed

Hypotheses citing non-existent PMIDs → flagged for evidence review

Orphaned universal_artifacts (no links, 30+ days old) → archive proposed

Metrics logged to garden_health_log each cycle

No duplicate pending governance decisions accumulate across cycles

Pipeline Steps

Step	Name	Action
0	Gov Dedup Prune	Delete older duplicate pending decisions (keep newest per content_id+decision_type)
1	Stale Gap Detection	Flag open gaps with no analyses/debates for 7+ days
2	Dedup Auto-Approval	Auto-approve+execute merges with similarity ≥ 0.95
3	Quality Enforcement	Propose deprecation for unverified hypotheses with score < 0.4 for 3+ days
4	Citation Integrity	Flag hypotheses with PMIDs not in papers table
5	Orphan Cleanup	Archive artifacts with no incoming links after 30 days
6	Metrics Logging	Log 6 distribution metrics to garden_health_log

Tables Used

knowledge_gaps — stale gap detection
hypotheses — quality enforcement, citation integrity
dedup_recommendations — dedup auto-approval
universal_artifacts — orphan cleanup
governance_decisions — proposal store (also dedup-pruned each cycle)
garden_health_log — metrics output
artifact_links, knowledge_edges — orphan detection
papers — citation validity check

Work Log

2026-04-12 — Run 1 (initial creation)

Agent: sonnet-4.6 Branch: resume-20260412_011041_worktree Commit: 1cfc044f

Created scripts/garden_health_maintenance.py with full 6-step pipeline
Implemented batch PMID lookup in citation check (1 query vs N×M)
First run: 349 hypotheses checked, 179 with invalid PMIDs flagged

2026-04-12 — Run 2 (governance dedup fix)

Agent: sonnet-4.6 Branch: senate-run44-clean

Problem found: governance_decisions table has no UNIQUE constraint on (content_id, decision_type). The propose_governance_decision function relied on a constraint that doesn't exist, so every 6h cycle created new duplicate pending decisions. After 8 cycles: 1500 pending decisions with 7-8 copies per artifact.

Fixes applied:

propose_governance_decision — added explicit SELECT check for existing pending decision before INSERT, preventing future accumulation

Added prune_duplicate_governance_decisions() (Step 0) — runs first each cycle to clean up any stragglers; deletes all but newest pending decision per (content_id, decision_type)

Added --gov-dedup CLI flag for standalone invocation

Run results:

181 duplicate groups found, 1,236 duplicate decisions deleted
Pending decisions reduced: 1500 → 264 (all unique)
1 stale unverified hypothesis (quality enforcement)
179/349 hypotheses with PMIDs not in papers table (evidence review)
6 metric entries logged to garden_health_log

Next cycle: With the fix in place, no new duplicates should accumulate. The 179 citation flags are legitimate — PMIDs that haven't been fetched into the local papers table yet.

2026-04-12 — Run 3 (minimax:57)

Agent: minimax:57 Branch: task-810043b5-27ba-4055-8130-c640ed848400

Results:

Gov dedup prune: 0 groups, 0 deleted (fix holding — no new accumulation)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending (already processed in prior runs)
Quality: 1 stale unverified hypothesis (already has pending deprecation decision)
Citation integrity: 355 checked, 184 with invalid PMIDs, 0 new flags (217 evidence_review decisions already pending from prior cycles)
Orphans: 0 found
Metrics: 6 entries logged to garden_health_log
universal_artifacts: 20,534 total, 19,849 active
Pending governance decisions: 50 archive, 2 deprecate, 217 evidence_review

Status: Pipeline is healthy. No duplicate accumulation. All systems operating as expected.

2026-04-12 — Run 4 (sonnet-4.6:72)

Agent: sonnet-4.6:72 Branch: senate/garden-health-run4

Results:

Gov dedup prune: 0 groups, 0 deleted (fix holding — no new accumulation)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending
Quality: 1 stale unverified hypothesis (already has pending deprecation decision)
Citation integrity: 364 checked, 192 with invalid PMIDs, 8 new flags (new hypotheses added since run 3)
Orphans: 0 found
Metrics: 6 entries logged to garden_health_log

Status: Pipeline healthy. 8 new citation evidence_review decisions created for newly added hypotheses with unverified PMIDs.

2026-04-12 — Run 10 (sonnet-4.6:73)

Agent: sonnet-4.6:73 Branch: push-dedup-run10

Results:

Gov dedup prune: 0 groups, 0 deleted (fix holding — no accumulation)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending
Quality: 1 stale unverified hypothesis (already has pending deprecation decision)
Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
Orphans: 0 found
Metrics: 6 entries logged to garden_health_log

Status: Pipeline healthy and stable. No new issues. All six steps completed without errors.

2026-04-12 — Run 11 (sonnet-4.6:73)

Agent: sonnet-4.6:73 Branch: wiki-edit-market-c6bd6dc1

Results:

Gov dedup prune: 0 groups, 0 deleted (fix holding — no accumulation)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending
Quality: 1 stale unverified hypothesis (already has pending deprecation decision)
Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
Orphans: 0 found (universal_artifacts table empty for 30+ day window)
Metrics: 6 entries logged to garden_health_log

Status: Pipeline healthy and stable. No new issues. All six steps completed without errors.

2026-04-12 — Run 12 (sonnet-4.6)

Agent: sonnet-4.6 Branch: task-eb8867b4-garden-health

Results:

Gov dedup prune: 0 groups, 0 deleted (fix holding — no accumulation)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending
Quality: 1 stale unverified hypothesis (already has pending deprecation decision, 0 new proposals)
Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
Orphans: 0 found (no active artifacts in 30+ day window)
Metrics: 6 entries logged to garden_health_log

Status: Pipeline healthy and stable. No new issues. All six steps completed without errors.

2026-04-12 — Run 13 (sonnet-4.6:73)

Agent: sonnet-4.6:73 Branch: orchestra/task/57531385-1f09-4e0d-ad35-fc45a509b0ee

Results:

Gov dedup prune: 0 groups, 0 deleted (no accumulation — fix continues to hold)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending
Quality: 1 stale unverified hypothesis found; existing pending deprecation decision prevented duplicate (idempotency working)
Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all 192 already have pending evidence_review decisions)
Orphans: 0 active artifacts in 30+ day window
Metrics: 6 entries logged to garden_health_log

Status: Pipeline healthy and stable. All six steps completed without errors in ~0.5s. Idempotency guards operating correctly across all steps.

2026-04-12 — Run 14 (sonnet-4.6:71)

Agent: sonnet-4.6:71 Branch: orchestra/task/eb8867b4-e8ed-4d36-82e1-3535919bd1e6

Results:

Gov dedup prune: 0 groups, 0 deleted (no accumulation)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending
Quality: 1 stale unverified hypothesis found; existing pending deprecation decision prevented duplicate (idempotency working)
Citation integrity: 364 checked, 192 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
Orphans: 0 active artifacts in 30+ day window
Metrics: 6 entries logged to garden_health_log

Status: Pipeline healthy and stable. All six steps completed without errors. No new issues.

2026-04-12 — Run 15 (sonnet-4.6)

Agent: sonnet-4.6 Branch: orchestra/task/6ae1c418-6f67-436a-a504-8072f9cacad0

Results:

Gov dedup prune: 0 groups, 0 deleted (no accumulation — fix continues to hold)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending
Quality: 0 stale unverified hypotheses (all previously flagged already have pending decisions)
Citation integrity: 373 checked (+9 new hypotheses), 201 with invalid PMIDs, 9 new flags (new hypotheses added since run 14)
Orphans: 0 active artifacts in 30+ day window
Metrics: 6 entries logged to garden_health_log

Status: Pipeline healthy and stable. 9 new evidence_review governance decisions created for newly added hypotheses with unverified PMIDs (h-var-a4975bdd96, h-var-b7e4505525, h-var-e95d2d1d86, h-var-bed9f3b7ed, h-var-18aae53fe9, h-var-f354525002, h-var-5aec85b987, h-var-2041072461, h-var-93e3ef09b3).

2026-04-12 — Run 16 (sonnet-4.6)

Agent: sonnet-4.6 Branch: senate/garden-run15-clean

Results:

Gov dedup prune: 0 groups, 0 deleted (no accumulation — fix continues to hold)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending
Quality: 0 stale unverified hypotheses
Citation integrity: 373 checked, 201 with invalid PMIDs, 0 new flags (all already have pending evidence_review decisions)
Orphans: 0 active artifacts in 30+ day window
Metrics: 6 entries logged to garden_health_log

Status: Pipeline healthy and stable. No new issues. All six steps completed without errors in ~0.3s.

2026-04-12 — Run 17 (sonnet-4.6)

Agent: sonnet-4.6 Branch: orchestra/task/46ae57a3-2883-4a36-a888-fc39d0812b27

Results:

Gov dedup prune: 0 groups, 0 deleted (no accumulation — fix continues to hold)
Stale gaps: 0 stale found
Dedup: 0 high-confidence pending
Quality: 0 stale unverified hypotheses
Citation integrity: 373 checked, 201 with invalid PMIDs, 0 new flags (all 234 already have pending evidence_review decisions)
Orphans: 0 active artifacts in 30+ day window (19,849 active UA, all created within last 30 days)
Metrics: 6 entries logged to garden_health_log

Status: Pipeline healthy and stable. No new issues. All six steps completed without errors in ~0.3s.

2026-04-13 — Run 18 (sonnet-4.6:41)

Agent: sonnet-4.6:41 Branch: task-b342c414-speclog-final Commit: 303cd9928

Bug found and fixed: auto_approve_dedup() handled merge_wiki_pages and merge_hypotheses but not merge_gaps. This caused high-confidence gap dedup approvals to be silently skipped (status set to 'approved' but no actual merge executed). A previous cycle had approved 14 merge_gaps records with no execution.

Fixes applied:

Added merge_gaps handler: resolves absorbed gaps (status='resolved', parent_gap_id=survivor) and marks their universal_artifact entries as merged.

Extended dedup query to also fetch already-approved merge_gaps records, so previously missed merges get executed in the next cycle without requiring a DB reset.

Added with_retry() helper with exponential backoff (up to 6 attempts) for write operations that hit DB lock errors from the concurrent API process.

Added PRAGMA busy_timeout=60000 to write connections.

Results:

Gov dedup prune: 0 groups, 0 deleted
Stale gaps: 0 stale found
Dedup: 14 merge_gaps executed (previously stuck approved-but-unexecuted)
Quality: 0 stale unverified
Citation integrity: 379 checked, 206 with invalid PMIDs, 1 new flag
Orphans: 0 found
Metrics: 6 entries logged

Status: Bug fixed. 14 duplicate gaps (mouse-brain and SEAAD variant groups) now resolved in knowledge_gaps table with parent_gap_id pointers.

2026-04-18 — Run 27 (minimax:63)

Agent: minimax:63 Branch: orchestra/task/eb8867b4-knowledge-garden-automated-pruning-dedup

Problem found: The script was deprecated in commit b4a034242 ("Consolidate file naming: drop _pg suffix, deprecate 128 SQLite scripts") because it was SQLite-only and PG became the only backend.

Fixes applied:

Re-created scripts/garden_health_maintenance.py as PostgreSQL-compatible version

Changed all SQL placeholders from ? (SQLite) to %s (PostgreSQL)

Used get_connection() from scidex.core.db_connect for database access

Fixed datetime comparison in enforce_quality(): compared datetime object with string cutoff

Fixed JSON handling in check_citation_integrity(): PostgreSQL JSONB returns parsed list, not string

Fixed COUNT(*) column access: use fetchone()["cnt"] instead of fetchone()[0] for dict_row

Fixed evidence_for empty string comparison in PostgreSQL: used evidence_for::text != '' instead of evidence_for != ''

Fixed garden_health_log sequence sync issue: nextval was returning existing IDs

Results:

Gov dedup prune: 0 groups, 0 deleted
Stale gaps: 2977 stale found, governance decisions proposed (idempotent — skips if already pending)
Dedup: 1 high-confidence approved and executed
Quality: 49 stale unverified hypotheses proposed for deprecation
Citation integrity: 591 checked, 7 with invalid PMIDs flagged for evidence review
Orphans: 0 found
Metrics: 6 entries logged to garden_health_log

Status: Pipeline restored and working. All six steps completed without errors.

Tasks using this spec (1)

[Senate] Knowledge garden — automated pruning, dedup, qualit

open P88

File: eb8867b4_e8e_spec.md

Modified: 2026-04-25 23:40

Size: 15.0 KB