[Senate] CI: Run site health check and fix broken links

[Senate] CI: Run site health check and fix broken links

> ## Continuous-process anchor
>
> This spec describes an instance of one of the retired-script themes
> documented in docs/design/retired_scripts_patterns.md. Before
> implementing, read:
>
> 1. The "Design principles for continuous processes" section of that
> atlas — every principle is load-bearing. In particular:
> - LLMs for semantic judgment; rules for syntactic validation.
> - Gap-predicate driven, not calendar-driven.
> - Idempotent + version-stamped + observable.
> - No hardcoded entity lists, keyword lists, or canonical-name tables.
> - Three surfaces: FastAPI + orchestra + MCP.
> - Progressive improvement via outcome-feedback loop.
> 2. The theme entry in the atlas matching this task's capability:
> S2, S4 (pick the closest from Atlas A1–A7, Agora AG1–AG5,
> Exchange EX1–EX4, Forge F1–F2, Senate S1–S8, Cross-cutting X1–X2).
> 3. If the theme is not yet rebuilt as a continuous process, follow
> docs/planning/specs/rebuild_theme_template_spec.md to scaffold it
> BEFORE doing the per-instance work.
>
> **Specific scripts named below in this spec are retired and must not
> be rebuilt as one-offs.** Implement (or extend) the corresponding
> continuous process instead.

Quest: Senate Priority: P80 Status: open

Goal

CI: Run site health check and fix broken links

Context

This task is part of the Senate quest (Senate layer). It contributes to the broader goal of building out SciDEX's senate capabilities.

Acceptance Criteria

Approach

Read relevant source files to understand current state

Plan implementation based on existing architecture

Implement changes

Test affected pages with curl

Commit with descriptive message and push

Work Log

2026-04-04 05:31 PDT — Slot 8

• Started recurring run for site health verification and broken-link triage.
• Read task spec and prepared to run status/page/link checks with command timeouts.
• Ran timeout 120 scidex status: API and nginx healthy; core services responsive.
• Ran key route checks on FastAPI (/, /exchange, /gaps, /graph, /analyses/, /atlas.html, /how.html, /senate, /forge) and observed 200/301/302 responses.
• Ran timeout 300 python3 link_checker.py; crawl hit transient connection-refused events during API cycling and produced systemic false positives.
• Verified flagged routes manually; core pages returned 200 after retries.
• Found actionable issue: artifact pages could render /hypothesis/{id} links from artifact metadata even when that hypothesis no longer exists.
• Updated api.py to only create hypothesis links when the target exists; otherwise render a non-clickable "missing" badge for context.
• Verified syntax with python3 -c "import py_compile; py_compile.compile('api.py', doraise=True)".
• Verified behavior with FastAPI TestClient on /artifacts: stale links to h-075f1f02 and h-seaad-51323624 are absent.
• Result: reduced real broken-link surface by preventing stale hypothesis links from being emitted.

2026-04-04 12:32 UTC — Slot 4

• Checked 30 routes manually: all return 200/301/302
• Routes checked: /, /exchange, /gaps, /graph, /analyses/, /demo, /showcase, /notebooks, /artifacts, /targets, /senate, /forge, /search, /agents, /quests, /demo/showcase (301→/showcase), /demo/walkthrough (301→/demo), /atlas.html, /market, /leaderboard, /challenges, /missions, /wiki, /papers, /entity/APOE, /resources, /status, /vision, /compare, /how.html
• No broken links found — site is healthy
• Result: CI check passed. All main routes healthy.

2026-04-04 04:25 PT — Slot 2

• Ran scidex status: API healthy, 76 analyses, 180 hypotheses
• Checked all key pages: /, /exchange, /gaps, /graph, /analyses/, /senate, /forge → all 200/302
• Ran timeout 300 python3 link_checker.py: crawled 2692 pages, found 21673 broken link references
• Root cause: link checker ran while API briefly restarted (saw "Connection refused" errors mid-run)

- All 500 errors (/agora, /challenges, /missions, /experiments, /agents, /wiki, /senate, /compare) were transient — manual checks return 200
- /image (HTTP 0, 15 links) = data:image/png;base64 URIs misidentified as URLs by link checker

• Closed 8 false-positive tasks created by link checker
• Result: Site is healthy. No real broken links found.

2026-04-04 (Slot 2, second run)

Routes Checked (27 total):

• All return 200/302 — no 500 errors
• /walkthrough.html was returning 404 — fixed by adding redirect route (task cb52ff0c)
• /notebooks responds slowly (>15s) but returns 200 — performance issue, not a broken link

Notable fixes from this session:

/walkthrough.html 404 → fixed with 301 redirect to /walkthrough

/artifacts 500 (UnboundLocalError: caption) → fixed by moving caption init before if block

/target/{id} 500 (TypeError: str vs int) → fixed by int() conversion

/agents, /agents/performance 500 → fixed by creating missing DB tables

Closed 5 stale linkcheck tasks (target/agents/senate issues now resolved)

Result: All 27 main routes healthy. No unresolved broken links.

2026-04-04 (Slot 2, third run) — Site Health Check

API Status: API was found DOWN. Restarted with sudo systemctl start scidex-api.service.

Routes Checked (16 key routes):

• All return 200/301/302 after restart: /, /vision, /exchange, /analyses/, /gaps, /senate, /agora, /wiki, /hypotheses, /targets, /artifacts, /showcase, /demo, /notebooks, /style.css, /walkthrough.html

Fixes Applied: None — site is healthy. API restart was sufficient.

Result: ✅ Complete — All routes healthy after API restart. No broken links found.

2026-04-06 18:54 PDT — Slot (task:e6e1fc6a)

• Checked scidex status: API+nginx active, 308 hypotheses, 688K KG edges.
• Verified 29 core routes via HTTP (all 200/301/302 on localhost:8000).
• Ran targeted link extraction from /gaps, /exchange, /senate, /forge, /arenas pages.
• Found real broken links: /senate/agent/theorist, /senate/agent/skeptic, /senate/agent/synthesizer, /senate/agent/domain_expert all returning HTTP 500.
• Root cause: senate_agent_detail() (api.py:28618) crashes with TypeError: 'NoneType' object is not subscriptable when txn['created_at'] is NULL in token_ledger rows. Secondary: txn['balance_after'] also NULL, crashing :, format.
• Fix: Guard all three nullable fields — reason, created_at, balance_after — with or ''/or 0 defaults (api.py lines 28615-28627).
• Deployed: two commits pushed via orchestra sync push, API restarted.
• Verified: all 4 senate/agent routes now return 200.
• All other checked routes (30+) remain healthy.

Result: ✅ Fixed — 4 senate/agent detail pages were returning 500 due to NULL token history fields; patched and deployed.

2026-04-12 11:20 UTC — Slot (task:e6e1fc6a)

• Checked 35+ routes manually via curl (all return 200/301/302/307).
• Routes confirmed healthy: /, /exchange, /gaps, /graph, /analyses/, /forge, /senate, /agora,

/atlas, /wiki, /papers, /notebooks, /artifacts, /targets, /agents, /quests, /leaderboard,
/challenges, /missions, /market, /economy, /search, /compare, /demo, /showcase, /status,
/vision, /clinical-trials, /experiments, /benchmarks, /arenas, /entity/APOE, /walkthrough

• Found: /datasets returns 404 — this route was never implemented; the Datasets nav link

correctly uses /artifacts?artifact_type=dataset. Removed datasets from DYNAMIC_ROUTES
in link_checker.py to eliminate false positive in future checker runs.

• Found: /resources responds slowly (~41s) but returns 200 — performance issue only.
• Fixed link_checker.py SITE_DIR: was hardcoded to a stale worktree path; changed to

Path(__file__).parent / "site" so the checker always uses its own worktree. Result: ✅ Site healthy. Fixed link_checker.py portability (SITE_DIR). Removed stale /datasets from DYNAMIC_ROUTES.

2026-04-12 12:00 UTC — Slot (task:e6e1fc6a)

• API running at 96% CPU (active inference workload); some routes take 30-60+s to respond.
• Verified all main routes via nginx (port 80): all return 301 (HTTP→HTTPS redirect) — healthy.
• Verified via HTTPS: /, /exchange, /forge, /market, /challenges, /showcase, /notebooks all 200.
• Routes /gaps, /senate, /leaderboard return 499 (client-closed-request) at 60s — confirmed via

nginx error log these DO respond successfully given enough time; 499 = our curl timing out, not
a server-side failure. Large page responses buffered to nginx tmp files (normal for big HTML).

• Ran link_checker.py on site/ static files: confirmed no broken static file references.
• Confirmed no broken internal href links in any static HTML in site/.
• Noted: /robots.txt returns 404 — crawlers (Applebot, etc.) are requesting it. Not a broken

link per se, but worth tracking. Result: ✅ Site healthy. All routes functional. Slow routes (/gaps, /senate, /leaderboard)
are performance-bound by CPU load — not broken links. No fixes needed this run.

2026-04-12 18:20 UTC — Slot (task:e6e1fc6a)

• API healthy: 264 analyses, 364 hypotheses, 700K+ KG edges, agent active.
• Verified 30+ routes via curl (localhost:8000): all return 200/301/302.

- Routes confirmed: /, /exchange, /gaps, /graph, /analyses/, /atlas.html, /how.html,
/senate, /forge, /demo, /showcase, /notebooks, /artifacts, /targets, /search,
/agents, /quests, /market, /leaderboard, /challenges, /missions, /wiki, /papers,
/resources, /status, /vision, /compare, /arenas, /entity/APOE, /walkthrough.html,
/agora, /experiments

• Ran link_checker.py on site/ static files: 508 HTML files checked, 0 broken links found.
• API data endpoints healthy: /api/hypotheses, /api/analyses, /api/wiki all return 200.
• No fixes needed this run.

Result: ✅ Site healthy. All routes return 200/301/302. No broken links found.

Tasks using this spec (1)

[Senate] CI: Run site health check and fix broken links

File: e6e1fc6a-69a_senate_ci_run_site_health_check_and_fix_spec.md

Modified: 2026-04-25 23:40

Size: 10.3 KB