> ## Continuous-process anchor
>
> This spec describes an instance of one of the retired-script themes
> documented in docs/design/retired_scripts_patterns.md. Before
> implementing, read:
>
> 1. The "Design principles for continuous processes" section of that
> atlas — every principle is load-bearing. In particular:
> - LLMs for semantic judgment; rules for syntactic validation.
> - Gap-predicate driven, not calendar-driven.
> - Idempotent + version-stamped + observable.
> - No hardcoded entity lists, keyword lists, or canonical-name tables.
> - Three surfaces: FastAPI + orchestra + MCP.
> - Progressive improvement via outcome-feedback loop.
> 2. The theme entry in the atlas matching this task's capability:
> S2 (pick the closest from Atlas A1–A7, Agora AG1–AG5,
> Exchange EX1–EX4, Forge F1–F2, Senate S1–S8, Cross-cutting X1–X2).
> 3. If the theme is not yet rebuilt as a continuous process, follow
> docs/planning/specs/rebuild_theme_template_spec.md to scaffold it
> BEFORE doing the per-instance work.
>
> **Specific scripts named below in this spec are retired and must not
> be rebuilt as one-offs.** Implement (or extend) the corresponding
> continuous process instead.
Test every registered FastAPI route (not just links found in HTML) for HTTP 500 errors every 8 hours. Catch orphaned routes, broken templates, and None-in-f-string errors before users report them.
The existing link checker (scidex-linkcheck service) crawls pages and checks links found in HTML content. But it misses routes like /agent-performance returning 500 because no page links to it directly (orphaned decorator with no function body).
This route-level health check complements the link checker by testing every FastAPI route directly.
api.py using app.routes — done via /openapi.json (avoids import issues)route_health table for dashboardingapp.routes to get all registered routes, filter to GET methods only, deduplicate{param} patterns with test for health check requestsSciDEX-RouteHealth/1.0journalctl -u scidex-api for recent traceback linesapi.py — FastAPI app with registered routesPostgreSQL — SQLite database with route_health tablescidex-api.service — journalctl source for tracebacksroute_health table — used by dashboard/CI reportinge6e1fc6a, 5616905a) — complementary, not overlappingcheck_route(): 2 retries on connection errors (curl returns 0), no retries on HTTP 500 (surfacing real bugs immediately)ci_route_health.py — found it already existed but had database locking issuesci_route_health.py:busy_timeout=30000 PRAGMA and retry logic for database locksget_recent_traceback() to look up journalctl for 500 error tracebackslog_health_check() for database locksextract_get_routes()--dry-run shows 338 routes, --limit 20 ran successfully with 1 HTTP 500 found on /network/discoveryscripts/ci_route_health.py missing from origin/main — lost during file-naming consolidation (b4a034242)/api/annotations + /api/annotations/{annotation_id}: Created migration 089 to add missing annotations table; applied to live DB/api/capsules/{capsule_id}/export + /api/capsules/{capsule_id}/export-files: Added missing import artifact_registry inline; fixed except HTTPException: raise to prevent 500 re-wrapping of 404s/api/content-owners/{artifact_id}: Removed ar.tier column from SELECT (column doesn't exist in agent_registry)/api/landscape/{domain}: Fixed query using disease/target_gene columns instead of non-existent domain column on hypotheses table/api/entity/{entity_name}: Changed p.pub_date → p.year (papers table uses year)
sqlite3.connect(Path("postgresql://scidex")) — invalid SQLite path; DB connection always failedscripts/ci_route_health.py:sqlite3.connect() with from scidex.core.database import get_db (PostgreSQL via psycopg)sys.path so scidex module resolvesTIMEOUT_SEC from 8s to 30s — many 500-returning routes are slow; 8s was causing curl timeouts before status code arrivedlog_result() INSERT to use explicit id subquery (avoids PostgreSQL SERIAL sequence-sync issues)init_table() to use PostgreSQL types (SERIAL, TIMESTAMPTZ) for CREATE TABLE IF NOT EXISTS
/api/agent-performance: SQL query had GROUP BY only on ph.hypothesis_id but selected h.title/h.composite_score; HAVING referenced alias events instead of COUNT(*) >= 2/api/backprop/status: PostgreSQL column detection code was incomplete migration stub (row[1].lower() with commented-out loop), causing UnboundLocalError/api/pools + /api/pools/leaderboard: datetime objects from pool rows not JSON serializable — added _serialize_pool() helperget_top_performers: GROUP BY only cp.id but selected non-aggregated pp.* columns — used subquery to get latest snapshot per pool/api/funding/summary: GROUP BY missing ar.name/cf.strategy; Decimal not JSON serializable — added _sanitize_funder_row() helper
/network/, /senate/ HTML pages) need separate investigationevidence_for = '' comparison against jsonb column — psycopg rejected the text-literal comparison on rows where evidence_for is null; fixed by replacing evidence_for = '' with evidence_for::text = '' at all 7 locations in api.py/network/ecosystem + /api/agents/ecosystem/overview: ecosystem_overview() in participant_contributions.py called suggest_nomination_actions() which closes the shared thread-local DB connection, invalidating the outer conn variable/api/edits: json.loads(row['diff_json']) called on value already parsed as dict by dict_row row_factory/api/entities: GROUP BY entity in outer query but entity_type in SELECT without aggregate — GroupingError/api/gaps/funding/stats: PostgreSQL doesn't allow aggregate alias in ORDER BY of same SELECT; also kg.title/kg.status not in GROUP BY/api/hypotheses/{hypothesis_id}/evidence + /api/hypotheses/{hypothesis_id}/evidence/provenance-mermaid: trust_score column doesn't exist in evidence_entries table — used in ORDER BY clause/api/proposals/{proposal_id}/status: imported from wrong module (scidex.exchange.exchange instead of scidex.senate.schema_governance)/api/proposals/pending: get_pending_approvals function did not exist
participant_contributions.py: re-acquire conn after suggest_nomination_actions() returnsapi.py: handle dict already-parsed JSON in diff_json fieldapi.py: use ANY_VALUE(entity_type) in outer GROUP BY query for entitiesfunding_allocators.py: replace ORDER BY aggregate alias with full expression; use ANY_VALUE for title/statusevidence_provenance.py: replace ORDER BY trust_score DESC with ORDER BY created_at DESCapi.py: fix get_proposal_status import sourceexchange.py: implement get_pending_approvals() using senate_proposals table
cfea322f0 to orchestra/task/1771ac79-route-health-check-test-all-known-routesroute_health table/, /health, /api/health, and /api/status timed out.ci_route_health.py: PID scrape via ss -tlnp returned no pid= metadata on this host, so the watchdog could not kill the stuck API.MainPID from systemctl show scidex-api.service instead.scidex-route-health.timer cadence from every 8 hours to every 15 minutes so hung backends are recovered promptly.{
"completion_shas": [
"0652dc49e4dd852457c74d35711f7d6f4c68ba99",
"d611be734560882f323b87703b68e1dbf379f4e0",
"607dabafc5582dc91af73709a8400f2a21cfea4f",
"7fa2078b99a8ac781e1a5acfdc22fbcdb4fc1eff",
"5ab47e6d07c2205b5f2e7253d2b29d7167a89314",
"b959c4db9f954ace104f47be5392ddc951519336",
"a884ea87888d6ad0510b06bb8fab1b39e3b36e75",
"6baeec23566db4ba9082b698088f56eaf1738072",
"6311dece3e18220b2b6101d246a441ed3111903d",
"cd4b79cffca8af0f9c8013f93667f4b17d9c7992",
"a5495ab8005d16e634f6d3b4806105b70c5b47f1",
"1775a7c3bfd903c65da141e82dce1374485d649d",
"2258b59ef82e6f5b99561969d5fc3d876e735644",
"8abc1361c67cc4ea94cdd3a605154a70b21ac307"
],
"completion_shas_checked_at": "2026-04-12T23:11:45.131384+00:00",
"completion_shas_missing": [
"ae21acf57ec95bbc80598916384cb0674fbae1d4",
"ebed2e29b1d0c1e8c48f472f955fb943a702f6cd",
"5c0a25ec9362a01e4fff3e7e005178faad39e344",
"85dada9b4e046c610976c74883f6e82b28be0307",
"a0556b113ed552595b838cd384fb58a4c72a09c0",
"d6fa283ac9db3af3bade2c297b8b5006c6234334",
"a6f1a784ebc49230a373231a208eb047700485aa",
"7acc5e123608d425ad3953c12591f3de126dca58",
"58a497aa8d25868b53509be0f097163db888f0e2",
"1418ec657bfea1764485141cde898ea47e0b5fa6",
"95fa9481c0e2c04b1a078c70ebdbc74d281f610f",
"fbf4af673ba9f2546df471db9930c2093638d1da",
"b0caf209cb976511af06519481619049157a60c6",
"3ee013f3081e938d5b863a6c5c071dbcd553b92c",
"d98da97d87f9090a922cc2ba9f2111c8261a1f07",
"45dffbd7bcd6eb273b30d50fe3383aa2f532b198",
"f46115918a342bd8d1fc45f4cad4baa18c64bcb7",
"e61e397f042fd2b7eb855290242546426cf20e39",
"018611a51c2a3b012507b0c401e22a66395be0b2"
],
"requirements": {
"coding": 5
}
}