[Artifacts] CI: Verify notebook coverage and generate summaries for recent analyses open analysis:6 reasoning:6 safety:6

Goal

> ## Continuous-process anchor
>
> This spec describes an instance of one of the retired-script themes
> documented in docs/design/retired_scripts_patterns.md. Before
> implementing, read:
>
> 1. The "Design principles for continuous processes" section of that
> atlas — every principle is load-bearing. In particular:
> - LLMs for semantic judgment; rules for syntactic validation.
> - Gap-predicate driven, not calendar-driven.
> - Idempotent + version-stamped + observable.
> - No hardcoded entity lists, keyword lists, or canonical-name tables.
> - Three surfaces: FastAPI + orchestra + MCP.
> - Progressive improvement via outcome-feedback loop.
> 2. The theme entry in the atlas matching this task's capability:
> A6 (pick the closest from Atlas A1–A7, Agora AG1–AG5,
> Exchange EX1–EX4, Forge F1–F2, Senate S1–S8, Cross-cutting X1–X2).
> 3. If the theme is not yet rebuilt as a continuous process, follow
> docs/planning/specs/rebuild_theme_template_spec.md to scaffold it
> BEFORE doing the per-instance work.
>
> **Specific scripts named below in this spec are retired and must not
> be rebuilt as one-offs.** Implement (or extend) the corresponding
> continuous process instead.

Recurring CI task for Quest 17 (Rich Artifacts). Scan for completed analyses that lack notebooks, generate notebook stubs for the top 3 missing, verify /artifacts and /notebooks pages load correctly, and report coverage percentage to track progress toward 100% notebook coverage.

Acceptance Criteria

Approach

Query DB: SELECT id, title FROM analyses WHERE status='completed' AND id NOT IN (SELECT DISTINCT analysis_id FROM notebooks WHERE analysis_id IS NOT NULL)

Sort by composite_score desc, take top 3

Generate notebook stubs via post_process.py or direct DB insert with summary content

Verify artifact pages load with curl

Compute coverage: count(notebooks with analysis_id) / count(completed analyses)

Log results in work log

Dependencies

• Completed analyses in DB
• notebooks table with analysis_id column
• /artifacts page route in api.py

Dependents

• Demo showcase (Quest 16) depends on rich artifact coverage

Work Log

(timestamped entries added during execution)

2026-04-04 09:22 PT — Slot 0

• Started recurring Quest 17 notebook coverage CI run in worktree task-083db551-3597-46e6-94a0-714ea9087bb3.
• Reviewed spec acceptance criteria and inspected ci_notebook_coverage.py implementation.
• Next step: run bounded coverage generation, verify /artifacts and /notebooks routes, and record coverage snapshot.

2026-04-04 09:24 PT — Slot 0 Completion

• Executed bounded coverage check: timeout 300 python3 ci_notebook_coverage.py

- Coverage snapshot: 43 / 43 completed analyses covered (100.00%)
- Missing analyses found: 0
- Stub generation: not needed this cycle

• Verified service and route health:

- timeout 120 scidex status → api and nginx active
- 200 /api/status
- 200 /artifacts
- 200 /notebooks
- 200 /notebook/nb-SDA-2026-04-04-gap-senescent-clearance-neuro (existing notebook still accessible)

• Ran bounded link health check: timeout 300 python3 link_checker.py

- Crawl result: 1826 pages, 2064 links, 0 concrete broken links
- Transient restart/outage retries were auto-classified by checker; final report remained healthy.

• Result: ✅ Quest 17 notebook-coverage CI passed with full coverage and no route regressions.

2026-04-04 05:58 PT — Slot 0

• Started recurring CI execution for notebook coverage verification and stub generation.
• Pulled latest worktree state, checked scidex status, and read task spec + quest priorities.
• Next step: query notebook coverage gaps and generate top missing notebook stubs.

2026-04-04 06:06 PT — Slot 0

• Implemented ci_notebook_coverage.py for recurring Artifact CI coverage checks and stub generation.
• Coverage scan (canonical completed analyses): 35 / 41 covered = 85.37%.
• Generated notebook stubs for top 3 currently missing analyses:

- SDA-2026-04-04-gap-tau-prop-20260402003221
- astrocyte-subtypes
- gba-pd

• Verified routes:

- GET /api/status → 200
- GET /artifacts → 200
- GET /notebooks → 200
- Existing notebook detail GET /notebook/nb-SDA-2026-04-04-gap-senescent-clearance-neuro → 200

• Ran link checker with timeout: timeout 300 python3 link_checker.py → no persistent broken links (0 broken), transient local API outages auto-suppressed by checker.

2026-04-04 (Slot 2) — Notebook Coverage CI

• Ran python3 ci_notebook_coverage.py
• Coverage: 35/42 completed analyses = 83.33%
• Generated 3 new notebook stubs for uncovered analyses:

- SDA-2026-04-04-gap-neuro-microglia-early-ad-20260404
- SDA-2026-04-04-gap-tau-prop-20260402003221
- astrocyte-subtypes

• Verified routes: /artifacts → 200, /notebooks → 200, /notebook/nb-SDA-2026-04-04-gap-senescent-clearance-neuro → 200
• Result: ✅ CI complete — 83.33% coverage. 3 new stubs generated.

2026-04-04 06:09 PT — Slot 7

• Started recurring Quest 17 artifact coverage task after slot queue returned no eligible lease.
• Reviewed acceptance criteria and current notebook schema/routes in api.py.
• Plan for this run: compute missing notebook coverage from live DB, generate top 3 missing notebook stubs into site/notebooks/, verify /artifacts and /notebooks route health, and record coverage metrics.

2026-04-04 06:13 PT — Slot 7 Completion

• Added scripts/artifacts_notebook_coverage_ci.py to automate Quest 17 coverage checks and top-missing stub generation.
• Executed bounded run:

- timeout 180 python3 scripts/artifacts_notebook_coverage_ci.py --db-path postgresql://scidex --notebooks-dir site/notebooks --limit 3
- Coverage snapshot (DB-linked notebooks): 46/60 completed analyses (76.7%).
- Generated stubs for 3 missing completed analyses:
- SDA-2026-04-04-gap-neuro-microglia-early-ad-20260404
- SDA-2026-04-04-gap-tau-prop-20260402003221
- sda-2026-04-01-001

• Wrote new stub files under site/notebooks/ (.ipynb + .html for each analysis).
• Verification:

- timeout 120 scidex status => api and nginx active.
- curl checks after transient outage recovery:
- 200 /artifacts
- 200 /notebooks
- 200 /api/status
- 302 /
- 200 /notebooks/SDA-2026-04-04-gap-neuro-microglia-early-ad-20260404.html
- timeout 300 python3 link_checker.py completed with no broken-link report; run skipped deep crawl due repeated API readiness timeouts (transient service churn).

• Result: Artifact notebook coverage task advanced with reproducible CI tooling plus three new notebook stubs, while core artifact/notebook routes remained healthy.

2026-04-04 09:07 PT — Slot 2

• Started recurring Quest 17 notebook coverage CI execution from slot 2 after scheduler cooldown prevented lease assignment.
• Ran bounded coverage generation:

- timeout 300 python3 ci_notebook_coverage.py
- Coverage snapshot: 41 / 43 completed analyses covered (95.35%)
- Missing analyses found: 2 (below top-3 cap)
- Generated notebook stubs:
- gut-brain-ad
- trem2-ad
- Created files:
- site/notebooks/gut-brain-ad.ipynb
- site/notebooks/gut-brain-ad.html
- site/notebooks/trem2-ad.ipynb
- site/notebooks/trem2-ad.html

• Verified service/routes:

- timeout 120 scidex status → api active, nginx active
- 200 /api/status
- 200 /artifacts
- 200 /notebooks
- 200 /notebook/nb-SDA-2026-04-04-gap-senescent-clearance-neuro (existing notebook route still healthy)

• Ran bounded link check:

- timeout 300 python3 link_checker.py
- Crawl summary: 1584 pages, 1737 links, 8 broken refs; most failures were auto-classified as transient outage retries
- Report: logs/link-check-latest.json

• Result: ✅ Coverage verification and stub generation complete for this cycle; notebook coverage is now tracked at 95.35% with no regression on core artifact routes.

2026-04-04 10:53 PT — Slot 1

• Started recurring Quest 17 notebook coverage CI from slot 1 after claiming unassigned task.
• Ran bounded coverage check: timeout 300 python3 ci_notebook_coverage.py

- Coverage after (strict files): 100.00% (46/46 completed analyses covered)
- 3 stale refs repaired (neuroinflammation-microglial, gap-20260404-060512, gap-20260404-052358)
- Generated 3 new notebook stubs with artifact links

• Verified routes:

- 200 /api/status
- 200 /artifacts
- 200 /notebooks
- 200 /notebook/nb-SDA-2026-04-04-gap-senescent-clearance-neuro (existing notebook still accessible)

• Ran bounded link check: timeout 300 python3 link_checker.py

- Crawl: 3131 pages, 3265 links, 0 broken (after transient-outage reconciliation)
- Report: logs/link-check-latest.json

• Result: ✅ CI passed — 100% notebook coverage, all routes healthy, 0 broken links.

2026-04-10 15:44 UTC — Slot (task-bc5704ce)

• Started recurring Quest 17 notebook coverage CI from worktree task-bc5704ce-f2df-401b-a640-f57ded341ced.
• Previous commit (8e893dbe) included scope creep: hardcoded DB_PATH, removed participant_contributions.py + 4 API endpoints + migration + script, deleted 3 spec files, added .ipynb files to repo.
• Reverted all scope creep changes before committing:

- Restored agent_registry.py DB_PATH env-var fallback
- Restored api.py participant contribution endpoints
- Restored participant_contributions.py, migrations/060_agent_contribution_ledger.py, scripts/register_agent_finding.py
- Restored 3 deleted spec files (agent_ecosystem_activation, contributor_network_quest, quest_market_participants)

• Ran bounded coverage check: timeout 300 python3 ci_notebook_coverage.py

- Coverage after: 42.40% (53/125 completed analyses with notebooks), up from 40.00%
- 3 new stubs generated for missing analyses: SDA-2026-04-04-gap-epigenetic-reprog-b685190e, SDA-2026-04-04-analysis_sea_ad_001, SDA-2026-04-09-gap-debate-20260409-201742-ca7016f1
- Missing refs reduced: 75 → 72

• Verified routes: 200 /api/status, 200 /artifacts, 200 /notebooks
• Committed clean diff: daad56e4
• Result: ✅ Scope creep reverted, coverage improved to 42.40%, routes verified.

2026-04-10 15:39 UTC — Slot (task-bc5704ce)

• Started recurring Quest 17 notebook coverage CI from worktree task-bc5704ce-f2df-401b-a640-f57ded341ced.
• Ran bounded coverage check: timeout 300 python3 ci_notebook_coverage.py --no-register

- Coverage: 125 completed analyses, 50 with refs (40.00%), 50 with strict files (40.00%)
- Missing refs: 75 analyses still lack notebook coverage
- Stale refs: 5 (2 notebook_path + 3 notebook link)

• Generated 8 notebook stubs (3 missing_ref + 5 stale_ref repairs):

- SDA-2026-04-04-gap-epigenetic-reprog-b685190e (missing_ref, score 0.5609)
- SDA-2026-04-04-analysis_sea_ad_001 (missing_ref, score 0.5454)
- SDA-2026-04-09-gap-debate-20260409-201742-ca7016f1 (missing_ref, score 0.0000)
- SDA-2026-04-08-gap-pubmed-20260406-062218-5c7f15f4 (stale_ref)
- SDA-2026-04-08-gap-pubmed-20260406-062111-db808ee9 (stale_ref)
- SDA-2026-04-08-gap-debate-20260406-062033-fecb8755 (stale_ref)
- SDA-2026-04-03-gap-seaad-v4-20260402065846 (stale_ref)
- SDA-2026-04-03-gap-seaad-v2-20260402032945 (stale_ref)

• DB registration failed due to sqlite3.DatabaseError: database disk image is malformed in artifacts table INSERT — DB corruption confirmed by PRAGMA integrity_check. Registration skipped this cycle; stubs written only to worktree site/.
• Verified routes: 200 /api/status, 200 /artifacts, 200 /notebooks
• Stubs written to site/notebooks/ in worktree (will sync via orchestra push)
• Committed and pushed: 7d1532d1
• Result: ⚠️ Coverage at 40.00% — regression from prior runs (100% on Apr 4). DB corruption prevents registration; 75 analyses still missing coverage. Stale refs repaired, new stubs generated.

2026-04-06 11:56 UTC — Orphaned worktree (task-b4c60959)

• Started recurring Quest 17 notebook coverage CI.
• Ran: timeout 300 python3 ci_notebook_coverage.py
• Coverage snapshot:

- Completed analyses: 78 (up from 47 on 2026-04-04)
- Covered (reference): 58 → after stubs: 61 = 78.21%
- Missing refs: 20 (new analyses from Apr 2–6 not yet covered)

• Generated 3 new notebook stubs:

- SDA-2026-04-03-gap-aging-mouse-brain-v2-20260402
- SDA-2026-04-06-gap-pubmed-20260406-041428-e14e6524
- SDA-2026-04-04-gap-debate-20260403-222510-20260402

• Verified routes: 200 /api/status, 200 /artifacts, 200 /notebooks
• Result: ⚠️ Coverage at 78.21% — regression from 100% due to 31 new analyses added since Apr 4. Stubs generated for 3; 17 remaining.

2026-04-04 11:30 PT — Slot 1

• Started recurring Quest 17 notebook coverage CI run.
• Ran bounded coverage check: timeout 300 python3 ci_notebook_coverage.py --no-register

- Coverage before: 97.87% (45/47 strict files, 46/47 reference)
- Missing: SDA-2026-04-04-gap-apoe4-lipid-metabolism (1 analysis without notebook ref)
- Stale: SDA-2026-04-04-gap-20260404-052358 (stale notebook_path ref)

• Fixed DB lock issue in ci_notebook_coverage.py: added PRAGMA journal_mode=WAL and PRAGMA busy_timeout=30000 to _connect() function.
• Manual DB registration for missing analysis: inserted notebook, notebook artifact, and analysis artifact records directly via Python.
• Coverage after: 100.00% (47/47)
• Verified routes:

- 200 /api/status
- 200 /artifacts
- 200 /notebooks

• Running bounded link check: timeout 300 python3 link_checker.py (background)
• Result: ✅ CI advanced — 100% notebook coverage achieved, all routes healthy.

2026-04-12 15:40 UTC — Slot 72

• Started recurring Quest 17 notebook coverage CI from worktree task-685ade6e-b77e-4794-a6f8-cf2e21e0780c.
• Ran bounded coverage check: timeout 120 python3 ci_notebook_coverage.py --db-path postgresql://scidex --site-root /home/ubuntu/scidex/site

- Completed analyses: 75 (up from 93 on Apr 11 — some pruned/merged)
- Coverage before (reference): 65.33% (49/75)
- Coverage before (strict files): 65.33%
- Missing refs: 26 analyses lack notebook coverage
- Stale refs: 0

• Generated 3 new notebook stubs:

- SDA-2026-04-03-gap-immune-atlas-neuroinflam-20260402 (missing_ref, score 0.5220, hyps 1)
- SDA-2026-04-12-gap-debate-20260410-112908-13c403ee (missing_ref, score 0.4800, hyps 1)
- SDA-2026-04-04-frontier-immunomics-e6f97b29 (missing_ref, score 0.0000, hyps 0)

• Coverage after (reference): 69.33% (52/75)
• Coverage after (strict files): 69.33%
• Verified routes:

- 200 /api/status
- 200 /artifacts
- 200 /notebooks

• Reports written: logs/notebook-coverage-latest.json, logs/notebook-coverage-latest.md, site/notebooks/notebook_coverage_summary_latest.md
• Result: ✅ CI complete — 3 stubs generated, coverage 65.33%→69.33%, all routes healthy.

2026-04-11 08:11 PT — Slot 0

• Started recurring Quest 17 notebook coverage CI from worktree task-ef45217e-ef7d-4535-8cc9-d358abbf5258.
• Ran bounded coverage check: timeout 120 python3 ci_notebook_coverage.py --db-path postgresql://scidex --site-root /home/ubuntu/scidex/site

- Coverage before (reference): 48.39% (45/93)
- Coverage before (strict files): 0.00%
- Missing refs: 48 analyses lack notebook coverage
- Stale notebook_path refs: 22, Stale notebook link refs: 45

• Self-healing repair: generated and registered 48 notebook stubs (3 missing_ref + 45 stale_ref repairs), all with artifact links
• Top 3 newly generated stubs:

- SDA-2026-04-04-gap-epigenetic-reprog-b685190e (missing_ref, score 0.5204, hyps 7)
- SDA-2026-04-04-analysis_sea_ad_001 (missing_ref, score 0.5117, hyps 5)
- SDA-2026-04-11-gap-debate-20260410-111113-052488a8 (missing_ref, score 0.0000, hyps 0)

• Coverage after (strict files): 51.61% (48/93) — improved from 0.00%
• Verified routes:

- 200 /api/status
- 200 /artifacts
- 200 /notebooks

• Reports written: logs/notebook-coverage-latest.json, logs/notebook-coverage-latest.md, site/notebooks/notebook_coverage_summary_latest.md
• Result: ✅ CI complete — 48 stubs generated and registered, coverage up to 51.61% strict, all routes healthy.

2026-04-12 17:16 UTC — Slot minimax:55

• Started recurring Quest 17 notebook coverage CI from worktree task-730d0425-7980-4695-b216-6beac90b0603.
• Ran bounded coverage check: timeout 120 python3 ci_notebook_coverage.py --db-path postgresql://scidex --site-root /home/ubuntu/scidex/site

- Coverage before (reference): 69.33% (52/75 completed analyses)
- Coverage before (strict files): 69.33%
- Missing refs: 23 analyses lack notebook coverage
- Stale refs: 0

• Generated 3 new notebook stubs:

- SDA-2026-04-04-frontier-connectomics-84acb35a (missing_ref, score 0.0000, hyps 0)
- SDA-2026-04-04-frontier-proteomics-1c3dba72 (missing_ref, score 0.0000, hyps 0)
- SDA-2026-04-09-gap-debate-20260409-201742-ca7016f1 (missing_ref, score 0.0000, hyps 0)

• Coverage after (reference): 73.33% (55/75)
• Coverage after (strict files): 73.33%
• Verified routes (via port 8000):

- 200 /api/status
- 200 /artifacts
- 200 /notebooks

• Reports written: logs/notebook-coverage-latest.json, logs/notebook-coverage-latest.md, site/notebooks/notebook_coverage_summary_latest.md
• Result: ✅ CI complete — 3 stubs generated, coverage 69.33%→73.33%, all routes healthy.

2026-04-12 23:44 UTC — Cycle 43 Final Run

• Executed timeout 300 python3 ci_notebook_coverage.py in worktree task-33803258-84bd-4bde-873b-740f1de9d1c3
• Coverage before: 55/76 completed analyses (72.37% ref, 69.74% strict)
• Missing analyses found: 3 (stub candidates), 21 total without ref
• Stale refs repaired: 2 (SDA-2026-04-04-frontier-connectomics-84acb35a, SDA-2026-04-04-frontier-proteomics-1c3dba72)
• Generated 5 notebook stubs (3 missing_ref + 2 stale_ref repairs):

- SDA-2026-04-09-gap-debate-20260409-201742-1e8eb3bd (missing_ref)
- SDA-2026-04-09-gap-debate-20260409-201742-d279750b (missing_ref)
- SDA-2026-04-09-gap-debate-20260409-201742-5407d57d (missing_ref)
- SDA-2026-04-04-frontier-connectomics-84acb35a (stale_ref repair)
- SDA-2026-04-04-frontier-proteomics-1c3dba72 (stale_ref repair)

• Coverage after (reference): 76.32% (58/76)
• Coverage after (strict files): 76.32%
• Verified routes (via port 8000):

- 200 /api/status
- 200 /artifacts
- 200 /notebooks

• Reports: logs/notebook-coverage-latest.json, logs/notebook-coverage-latest.md, site/notebooks/notebook_coverage_summary_latest.md
• Result: ✅ CI complete — 5 stubs generated/repaired, coverage 72.37%→76.32%, all routes healthy.

2026-04-21 01:28 PT — Codex Slot 42

• Read shared Orchestra standards, local AGENTS.md, task spec, Quest 17 artifacts spec, artifact governance notes, and retired-script A6 continuous-process guidance.
• Ran scripts/ci_notebook_coverage.py --report-only --ci-mode: completed-analysis notebook references were 131/131 but strict file coverage was 76/131 (58.02%) with 20 stale analyses.notebook_path refs and 55 stale notebook-link refs.
• Found the self-healing driver could not register repairs on PostgreSQL because it still used SQLite INSERT OR REPLACE / INSERT OR IGNORE syntax; patched registration and artifact-link writes to PostgreSQL-compatible ON CONFLICT upserts and guarded inserts.
• Reran the coverage driver successfully: strict completed-analysis coverage is now 131/131 (100.00%), missing refs 0, stale analysis refs 0, stale notebook-link refs 0; 55 CI notebook stubs were registered and 57 stale work items were repaired/rewritten.
• Verified python3 -m py_compile scripts/ci_notebook_coverage.py passes.
• Route verification attempted for /api/status, /artifacts, /notebooks, and an existing notebook route, but the live API returned 500s from psycopg_pool.PoolTimeout before route-specific handling. scidex services restart api was attempted but sandbox sudo restrictions prevented an effective restart.