[Senate] CI: Run site health check and fix broken links blocked coding:8 safety:9

← Senate
Run link_checker.py on all main routes. Report broken links (true failures, not transient 502s). Fix any real issues found.

Completion Notes

Auto-release: recurring task had no work this cycle

Git Commits (20)

[Senate] CI health check: site healthy, all routes functional (performance-bound, not broken) [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-12
[Senate] CI health check: fix link_checker.py SITE_DIR and remove stale /datasets route [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-12
[Senate] Fix broken links: stop linking POST-only API endpoints as clickable hrefs [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-08
[Senate] Update spec with work log for site health run 2026-04-06 [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-06
[Senate] Fix senate/agent 500: guard None balance_after in token history [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-06
[Senate] Fix senate/agent 500 error: guard None created_at/reason in token history [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-06
[Senate] CI: Site health check — all 30 routes healthy [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-04
[Senate] CI: Site health check — restarted API, all 16 routes healthy [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-04
[Senate] Merge all slot work logs into health check spec [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-04
[Senate] CI: Site health check — all pages healthy, 2 minor doc gaps logged [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-04
[Senate] Guard artifact hypothesis links against missing IDs [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-04
[Senate] CI: Site health check — all 27 routes healthy [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-04
[Senate] Run site health check - confirmed service outage [task:e6e1fc6a]2026-04-04
[Senate] CI health check: site healthy, 8 false-positive link tasks closed [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-04
[Senate] Complete site health check - all pages working [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-03
[Senate] Fix entity page 500 errors with error handling2026-04-03
[Senate] Fix thread-safety in page cache and dict-type guard in mechanism infobox2026-04-02
[Senate] Add SQLite busy_timeout to prevent DB lock hangs [task:e6e1fc6a-69a2-49f4-aa66-7ef4c42cab1e]2026-04-02
[Senate] Fix nginx /analyses/ routing: fallback to FastAPI for missing static files2026-04-02
[UI] Fix wiki 500 error when tags contain dict objects instead of strings2026-04-02
Spec File

[Senate] CI: Run site health check and fix broken links

> ## Continuous-process anchor
>
> This spec describes an instance of one of the retired-script themes
> documented in docs/design/retired_scripts_patterns.md. Before
> implementing, read:
>
> 1. The "Design principles for continuous processes" section of that
> atlas — every principle is load-bearing. In particular:
> - LLMs for semantic judgment; rules for syntactic validation.
> - Gap-predicate driven, not calendar-driven.
> - Idempotent + version-stamped + observable.
> - No hardcoded entity lists, keyword lists, or canonical-name tables.
> - Three surfaces: FastAPI + orchestra + MCP.
> - Progressive improvement via outcome-feedback loop.
> 2. The theme entry in the atlas matching this task's capability:
> S2, S4 (pick the closest from Atlas A1–A7, Agora AG1–AG5,
> Exchange EX1–EX4, Forge F1–F2, Senate S1–S8, Cross-cutting X1–X2).
> 3. If the theme is not yet rebuilt as a continuous process, follow
> docs/planning/specs/rebuild_theme_template_spec.md to scaffold it
> BEFORE doing the per-instance work.
>
> **Specific scripts named below in this spec are retired and must not
> be rebuilt as one-offs.** Implement (or extend) the corresponding
> continuous process instead.

Quest: Senate Priority: P80 Status: open

Goal

CI: Run site health check and fix broken links

Context

This task is part of the Senate quest (Senate layer). It contributes to the broader goal of building out SciDEX's senate capabilities.

Acceptance Criteria

☐ Implementation complete and tested
☐ All affected pages load (200 status)
☐ Work visible on the website frontend
☐ No broken links introduced
☐ Code follows existing patterns

Approach

  • Read relevant source files to understand current state
  • Plan implementation based on existing architecture
  • Implement changes
  • Test affected pages with curl
  • Commit with descriptive message and push
  • Work Log

    2026-04-04 05:31 PDT — Slot 8

    • Started recurring run for site health verification and broken-link triage.
    • Read task spec and prepared to run status/page/link checks with command timeouts.
    • Ran timeout 120 scidex status: API and nginx healthy; core services responsive.
    • Ran key route checks on FastAPI (/, /exchange, /gaps, /graph, /analyses/, /atlas.html, /how.html, /senate, /forge) and observed 200/301/302 responses.
    • Ran timeout 300 python3 link_checker.py; crawl hit transient connection-refused events during API cycling and produced systemic false positives.
    • Verified flagged routes manually; core pages returned 200 after retries.
    • Found actionable issue: artifact pages could render /hypothesis/{id} links from artifact metadata even when that hypothesis no longer exists.
    • Updated api.py to only create hypothesis links when the target exists; otherwise render a non-clickable "missing" badge for context.
    • Verified syntax with python3 -c "import py_compile; py_compile.compile('api.py', doraise=True)".
    • Verified behavior with FastAPI TestClient on /artifacts: stale links to h-075f1f02 and h-seaad-51323624 are absent.
    • Result: reduced real broken-link surface by preventing stale hypothesis links from being emitted.

    2026-04-04 12:32 UTC — Slot 4

    • Checked 30 routes manually: all return 200/301/302
    • Routes checked: /, /exchange, /gaps, /graph, /analyses/, /demo, /showcase, /notebooks, /artifacts, /targets, /senate, /forge, /search, /agents, /quests, /demo/showcase (301→/showcase), /demo/walkthrough (301→/demo), /atlas.html, /market, /leaderboard, /challenges, /missions, /wiki, /papers, /entity/APOE, /resources, /status, /vision, /compare, /how.html
    • No broken links found — site is healthy
    • Result: CI check passed. All main routes healthy.

    2026-04-04 04:25 PT — Slot 2

    • Ran scidex status: API healthy, 76 analyses, 180 hypotheses
    • Checked all key pages: /, /exchange, /gaps, /graph, /analyses/, /senate, /forge → all 200/302
    • Ran timeout 300 python3 link_checker.py: crawled 2692 pages, found 21673 broken link references
    • Root cause: link checker ran while API briefly restarted (saw "Connection refused" errors mid-run)
    - All 500 errors (/agora, /challenges, /missions, /experiments, /agents, /wiki, /senate, /compare) were transient — manual checks return 200
    - /image (HTTP 0, 15 links) = data:image/png;base64 URIs misidentified as URLs by link checker
    • Closed 8 false-positive tasks created by link checker
    • Result: Site is healthy. No real broken links found.

    2026-04-04 (Slot 2, second run)

    Routes Checked (27 total):

    • All return 200/302 — no 500 errors
    • /walkthrough.html was returning 404 — fixed by adding redirect route (task cb52ff0c)
    • /notebooks responds slowly (>15s) but returns 200 — performance issue, not a broken link
    Notable fixes from this session:
  • /walkthrough.html 404 → fixed with 301 redirect to /walkthrough
  • /artifacts 500 (UnboundLocalError: caption) → fixed by moving caption init before if block
  • /target/{id} 500 (TypeError: str vs int) → fixed by int() conversion
  • /agents, /agents/performance 500 → fixed by creating missing DB tables
  • Closed 5 stale linkcheck tasks (target/agents/senate issues now resolved)
  • Result: All 27 main routes healthy. No unresolved broken links.

    2026-04-04 (Slot 2, third run) — Site Health Check

    API Status: API was found DOWN. Restarted with sudo systemctl start scidex-api.service.

    Routes Checked (16 key routes):

    • All return 200/301/302 after restart: /, /vision, /exchange, /analyses/, /gaps, /senate, /agora, /wiki, /hypotheses, /targets, /artifacts, /showcase, /demo, /notebooks, /style.css, /walkthrough.html
    Fixes Applied: None — site is healthy. API restart was sufficient.

    Result: ✅ Complete — All routes healthy after API restart. No broken links found.

    2026-04-06 18:54 PDT — Slot (task:e6e1fc6a)

    • Checked scidex status: API+nginx active, 308 hypotheses, 688K KG edges.
    • Verified 29 core routes via HTTP (all 200/301/302 on localhost:8000).
    • Ran targeted link extraction from /gaps, /exchange, /senate, /forge, /arenas pages.
    • Found real broken links: /senate/agent/theorist, /senate/agent/skeptic, /senate/agent/synthesizer, /senate/agent/domain_expert all returning HTTP 500.
    • Root cause: senate_agent_detail() (api.py:28618) crashes with TypeError: 'NoneType' object is not subscriptable when txn['created_at'] is NULL in token_ledger rows. Secondary: txn['balance_after'] also NULL, crashing :, format.
    • Fix: Guard all three nullable fields — reason, created_at, balance_after — with or ''/or 0 defaults (api.py lines 28615-28627).
    • Deployed: two commits pushed via orchestra sync push, API restarted.
    • Verified: all 4 senate/agent routes now return 200.
    • All other checked routes (30+) remain healthy.
    Result: ✅ Fixed — 4 senate/agent detail pages were returning 500 due to NULL token history fields; patched and deployed.

    2026-04-12 11:20 UTC — Slot (task:e6e1fc6a)

    • Checked 35+ routes manually via curl (all return 200/301/302/307).
    • Routes confirmed healthy: /, /exchange, /gaps, /graph, /analyses/, /forge, /senate, /agora,
    /atlas, /wiki, /papers, /notebooks, /artifacts, /targets, /agents, /quests, /leaderboard,
    /challenges, /missions, /market, /economy, /search, /compare, /demo, /showcase, /status,
    /vision, /clinical-trials, /experiments, /benchmarks, /arenas, /entity/APOE, /walkthrough
    • Found: /datasets returns 404 — this route was never implemented; the Datasets nav link
    correctly uses /artifacts?artifact_type=dataset. Removed datasets from DYNAMIC_ROUTES
    in link_checker.py to eliminate false positive in future checker runs.
    • Found: /resources responds slowly (~41s) but returns 200 — performance issue only.
    • Fixed link_checker.py SITE_DIR: was hardcoded to a stale worktree path; changed to
    Path(__file__).parent / "site" so the checker always uses its own worktree. Result: ✅ Site healthy. Fixed link_checker.py portability (SITE_DIR). Removed stale /datasets from DYNAMIC_ROUTES.

    2026-04-12 12:00 UTC — Slot (task:e6e1fc6a)

    • API running at 96% CPU (active inference workload); some routes take 30-60+s to respond.
    • Verified all main routes via nginx (port 80): all return 301 (HTTP→HTTPS redirect) — healthy.
    • Verified via HTTPS: /, /exchange, /forge, /market, /challenges, /showcase, /notebooks all 200.
    • Routes /gaps, /senate, /leaderboard return 499 (client-closed-request) at 60s — confirmed via
    nginx error log these DO respond successfully given enough time; 499 = our curl timing out, not
    a server-side failure. Large page responses buffered to nginx tmp files (normal for big HTML).
    • Ran link_checker.py on site/ static files: confirmed no broken static file references.
    • Confirmed no broken internal href links in any static HTML in site/.
    • Noted: /robots.txt returns 404 — crawlers (Applebot, etc.) are requesting it. Not a broken
    link per se, but worth tracking. Result: ✅ Site healthy. All routes functional. Slow routes (/gaps, /senate, /leaderboard)
    are performance-bound by CPU load — not broken links. No fixes needed this run.

    2026-04-12 18:20 UTC — Slot (task:e6e1fc6a)

    • API healthy: 264 analyses, 364 hypotheses, 700K+ KG edges, agent active.
    • Verified 30+ routes via curl (localhost:8000): all return 200/301/302.
    - Routes confirmed: /, /exchange, /gaps, /graph, /analyses/, /atlas.html, /how.html,
    /senate, /forge, /demo, /showcase, /notebooks, /artifacts, /targets, /search,
    /agents, /quests, /market, /leaderboard, /challenges, /missions, /wiki, /papers,
    /resources, /status, /vision, /compare, /arenas, /entity/APOE, /walkthrough.html,
    /agora, /experiments
    • Ran link_checker.py on site/ static files: 508 HTML files checked, 0 broken links found.
    • API data endpoints healthy: /api/hypotheses, /api/analyses, /api/wiki all return 200.
    • No fixes needed this run.
    Result: ✅ Site healthy. All routes return 200/301/302. No broken links found.

    Payload JSON
    {
      "requirements": {
        "coding": 8,
        "safety": 9
      },
      "auto_tagged_at": "2026-04-03T22:29:52.509378",
      "completion_shas": [
        "fdd8e6ad643ed5dee7fb4a7bc0b55d3c404ff568",
        "7526cf07dda84d86e1d11680f65ef5b9f230dcf2"
      ],
      "completion_shas_checked_at": "2026-04-12T18:23:13.485853+00:00",
      "completion_shas_missing": [
        "64fe558f83929f325045740f0d4267ba8515c472",
        "bb80043d744abb94030e3730dd1a60d48651b832",
        "7e936384d3549cb5456b71d835cdd1bde58a20f6",
        "6e1ab4b04f17af50fd37d7dd26cbdf8420d9ff05",
        "cf9cf2cab7b4ae3a150c29414e982e00424684a5",
        "3d9af869ac8cde93decfd3c5a1e8e9cf6bd1bc5e",
        "6e2e86040414591ae7b8e56bc4529c8fa291599e",
        "1a034fff6564be6cbc5256fd4578a56e0e71d119",
        "0e77076471912acad0990608ab94b12fa00dd664",
        "3d360553b4632ae4bf1813c8f359a529e27f54bc",
        "12cdaea8e0f2e0d6e11c8a9663e9efd9a333d4de",
        "8c2039f33ed35461879b322ae36f72b9859b4b27",
        "f7025915f70930c858a64a87549303058da34ada",
        "a5a9fcb47806cc9002e88e5032874e2f40d7a826",
        "87602b39c927bf74e6c7291e142c40f80b25587c",
        "ca994924bfb7dc0c88e9e2b7d0a1ac237d226446",
        "ae75c7d29c6d6800ec875582bd3d7bebae22a82b",
        "1db999568a800c444550c1585ea539d0f147d572",
        "863423b1a9fc8c73d29b44e33d319e36a5f2c507",
        "9b7576b30ec0d26368bb67585751094f3a901d72",
        "ff9f2cda30c8f1b0a5953f06e94383c8a2ccd2f0",
        "dcc1a4c36e4f50b30948bc447cb6446b0131e382"
      ]
    }

    Sibling Tasks in Quest (Senate) ↗

    Task Dependencies

    ↓ Referenced by (downstream)