SciDEX — Task: [Search] CI: Verify search indexes are current

Rebuild theme S3 (FTS/vector index coverage verification). SPEC: docs/planning/specs/rebuild_theme_S3_search_index_coverage_spec.md. Template: docs/planning/specs/rebuild_theme_template_spec.md. Principles: docs/design/retired_scripts_patterns.md. Self-describing via pg_indexes introspection — no hardcoded index list. LLM only on rebuild-failure remediation. All other requirements from rebuild template.

Completion Notes

Auto-release: recurring task had no work this cycle

Git Commits (12)

[Cross-cutting] Search CI: indexes current, all 5 FTS tables in sync, recent content discoverable [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-12

[Search] Re-verify index freshness and endpoint health [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-04

[Search] Verify search indexes and rebuild FTS tables [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-04

[Search] Rebuild and verify FTS search indexes [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-04

[Search] Update search CI work log [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-04

[Search] CI: Verify search indexes are current - slot 8 re-verification [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-04

[Search] CI: Verify search indexes are current [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-04

[Search] CI: Search indexes verified current - 211 hypotheses, 17.3K wiki pages, 62 analyses [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-04

[Search] CI: Smart DB resolution for worktree search verification2026-04-04

[Search] Add busy_timeout to FTS5 scripts to prevent lock failures [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-02

[Search] Add CI script for search index verification and rebuild missing FTS5 indexes [task:3a897f8a-0712-4701-a675-07f0670d8f87]2026-04-02

Spec File

[Search] CI: Verify search indexes are current

> ## Continuous-process anchor
>
> This spec describes an instance of one of the retired-script themes
> documented in docs/design/retired_scripts_patterns.md. Before
> implementing, read:
>
> 1. The "Design principles for continuous processes" section of that
> atlas — every principle is load-bearing. In particular:
> - LLMs for semantic judgment; rules for syntactic validation.
> - Gap-predicate driven, not calendar-driven.
> - Idempotent + version-stamped + observable.
> - No hardcoded entity lists, keyword lists, or canonical-name tables.
> - Three surfaces: FastAPI + orchestra + MCP.
> - Progressive improvement via outcome-feedback loop.
> 2. The theme entry in the atlas matching this task's capability:
> S3 (pick the closest from Atlas A1–A7, Agora AG1–AG5,
> Exchange EX1–EX4, Forge F1–F2, Senate S1–S8, Cross-cutting X1–X2).
> 3. If the theme is not yet rebuilt as a continuous process, follow
> docs/planning/specs/rebuild_theme_template_spec.md to scaffold it
> BEFORE doing the per-instance work.
>
> **Specific scripts named below in this spec are retired and must not
> be rebuilt as one-offs.** Implement (or extend) the corresponding
> continuous process instead.

Quest: Search Priority: P82 Status: running Frequency: daily

Goal

Check if search results include recently added hypotheses, analyses, and wiki pages. Rebuild indexes if stale.

Approach

Query the database to check if recent items (added in last 24h) appear in search

Check hypotheses, analyses, and wiki_entities tables for recent entries

If items are missing from expected search results, flag for investigation

Verify the /api/search endpoint works correctly

Verification Steps

1. Check recent hypotheses

Query for hypotheses added in last 24 hours
Test if they appear in /api/search?q={term}

2. Check recent analyses

Query for analyses added in last 24 hours
Test if they appear in search

3. Check wiki_entities

Query for wiki_entities added/updated recently
Verify entity search works

4. API endpoint test

curl -s http://localhost:8000/api/search?q=test | python3 -m json.tool
Verify it returns valid JSON with expected structure

Work Log

2026-04-04 05:26 PDT — Slot 3

Claimed task via orchestra task get-next --slot 3 --project SciDEX
Reviewed search implementation in api.py (/api/search) and rebuild utility rebuild_fts.py
Verified index parity against live DB postgresql://scidex:

- hypotheses vs hypotheses_fts: 181 / 181, missing 0
- analyses vs analyses_fts: 77 / 77, missing 0
- wiki_pages vs wiki_pages_fts: 1786 / 1786, missing 0
- papers vs papers_fts: 1271 / 1271, missing 0
- wiki_entities_fts: table missing (not present in DB)

Ran timeout 300 python3 rebuild_fts.py postgresql://scidex:

- hypotheses_fts, analyses_fts, wiki_pages_fts, papers_fts rebuilt successfully
- wiki_entities_fts skipped because table does not exist

API service on systemd (/home/ubuntu/scidex) was unstable during run due unrelated import failure (ModuleNotFoundError: resource_tracker), so endpoint checks were executed against temporary worktree API instance on 127.0.0.1:8010
Verified /api/search returns healthy multi-type results:

- q=tau → 113 results
- q=Alzheimer → 121 results
- q=Microglial → 104 results

Verified latest content discoverability (recent hypotheses, analyses, wiki entities): exact-title hits were found for all sampled recent rows
Result: Search indexes are current for existing FTS tables; entity search currently relies on SQL fallback because wiki_entities_fts is absent

2026-04-04 05:30 UTC — Slot 4

Task claimed from Orchestra (was stale running)
Starting verification of search indexes

2026-04-04 05:35 UTC — Verification Complete

Database Status:

Table	Count	Latest Entry
hypotheses	211	2026-04-04T08:59:36
analyses	62	2026-04-04T08:59:36
wiki_entities	13,640	2026-04-03T06:14:52

Search API Tests:

/api/search?q=tau → 30 results, correct types (hypothesis, analysis, entity, etc.)
/api/search?q=Microglial → 37 results, most recent "Microglial TREM2 downregulation..." found
/api/search?q=Alzheimer → 97 results, 13 analysis results including SEA-AD
/api/search?q=parabrachial → 9 results, entity "parabrachial-nucleus-gustatory" found

Search Page:

/search → 200 OK

Conclusion: All search indexes are current. Recent hypotheses, analyses, and wiki_entities are all findable via search. No rebuild needed.

2026-04-04 (Slot 1) — Re-verification

Database Status:

Table	Recent (24h)	Latest Entry
hypotheses	12 new	2026-04-04T08:59:36
analyses	19 new	2026-04-04T08:59:36
wiki_entities	5,828 synced	2026-04-03T06:14:52

Search API Tests:

/api/search?q=tau → 140 results, types: hypothesis, analysis, entity, etc.
/api/search?q=Alzheimer → 168 results, all 10 types represented (hypothesis, analysis, entity, gap, notebook, debate, paper, wiki_page, target, experiment)
/search page → 200 OK

Conclusion: Search indexes are healthy and current. All recent content is indexed and searchable.

2026-04-04 11:00 UTC — Slot 8 Re-verification

Database Status:

Table	Count	Latest Entry
hypotheses	180	2026-04-02T19:54:17
analyses	76	2026-04-02T08:05:51
wiki_pages	1776	2026-04-02 07:20:14

Search API Tests:

/api/search?q=tau → 53 results (hypothesis, analysis, wiki_page, gap)
/api/search?q=Microglial → 26 results (hypothesis, analysis, wiki_page, gap)
/search page → 200 OK

Conclusion: Search indexes are healthy and current. All content is properly indexed and searchable via the API.

2026-04-04 05:24 PDT — Slot 8 Rebuild + Verification

Claimed task via orchestra task get-next --slot 8 --project SciDEX.
Ran timeout 300 env SCIDEX_DB=postgresql://scidex python3 scripts/ci_verify_search_indexes.py.
Detected stale/missing FTS artifacts:

- Missing tables: hypotheses_fts, wiki_pages_fts, papers_fts, analyses_fts
- Missing triggers: 0/12

Rebuilt indexes via migrate_fts5.py (triggered by CI script) against live DB.
Post-rebuild verification passed:

- hypotheses_fts = 181/181
- wiki_pages_fts = 1786/1786
- papers_fts = 1271/1271
- analyses_fts = 77/77
- 12/12 FTS triggers present

Verified /api/search health:

- curl -i http://localhost:8000/api/search?q=test → HTTP/1.1 200 OK
- Sampled latest hypotheses, analyses, and wiki pages from postgresql://scidex; all sampled titles were returned by /api/search (PASS).

Verified page render and service health:

- curl http://localhost:8000/search → 200
- scidex status → API and nginx active

Result: Search indexes were stale/missing and are now rebuilt/current; search endpoint returns valid results including recent content.

2026-04-04 10:56 PDT — Slot 5 Re-verification

Pulled latest branch state in worktree (git pull --rebase) before running CI checks.
Ran scidex status to capture service/database baseline:

- DB counts: analyses=83, hypotheses=255, knowledge_edges=1836.

Verified FTS index freshness with timeout guard:

- timeout 300 env SCIDEX_DB=postgresql://scidex python3 scripts/ci_verify_search_indexes.py
- Results: hypotheses_fts 255/255, wiki_pages_fts 1800/1800, papers_fts 1303/1303, analyses_fts 83/83, triggers 15/15.

Verified search endpoints and page render:

- curl http://localhost:8000/search → 200
- curl http://localhost:8000/api/search?q=tau → total_results=121, 9 content types.

Spot-checked recency discoverability:

- New hypothesis query YWHAG-Mediated returned 1/1 hit: YWHAG-Mediated TFEB Subcellular Targeting.
- New analysis query Neuroinflammation and microglial priming returned analysis hits including SDA-2026-04-04-gap-neuro-microglia-early-ad-20260404.

Observed transient API restart window (api_status=000 then recovered to 200 within ~10s); retried checks after service recovery.
Result: Search indexes are current and live search remains discoverable for recent content.

2026-04-12 14:15 UTC — Slot 71 Daily Verification

Database Status:

Table	Total	Recent (24h)	Latest Entry
hypotheses	349	14	2026-04-12 14:09:59
analyses	261	25	2026-04-12T07:09:58
wiki_pages	17539	21	2026-04-12T12:10:16
papers	15989	46	2026-04-12 14:09:54
wiki_entities	13640	N/A	—

FTS Index Parity:

Table	Main	FTS	Status
hypotheses	349	349	OK
analyses	261	261	OK
wiki_pages	17539	17539	OK
papers	15989	15989	OK
wiki_entities	13640	13640	OK

Search API Tests:

/api/search?q=tau → 183 results (hypothesis, wiki_page, etc.)
/api/search?q=NLRP3+Mitophagy → 25 results; recent hypothesis h-f10d82a7 returned as top hit ✓
/api/search?q=FcRn+Transport → 27 results; h-9d16afa0 (added 2026-04-12) returned ✓
/api/search?q=Self-Evolution+Loop → 1 result; wiki page self-evolution (added 2026-04-12) returned ✓
/search page → 200 OK

Conclusion: All FTS indexes are current (0 missing rows in all tables, including wiki_entities_fts now present). Recent hypotheses, analyses, and wiki pages from the last 24h are all discoverable via search. No rebuild needed.

Payload JSON

{
  "requirements": {
    "coding": 7,
    "reasoning": 6
  },
  "auto_tagged_at": "2026-04-03T22:29:52.510355",
  "completion_shas": [
    "5e558beb579108ba09160d969e61e8d5e8005664"
  ],
  "completion_shas_checked_at": "2026-04-12T18:02:34.524427+00:00",
  "completion_shas_missing": [
    "9228d4795b98e4f36964ffe46f3b7917de61bcf0",
    "9dfd06fddb421e742946a6ec6543ec3f82c4e6db",
    "f71f42dac02f6353a85b281f26933fa7d8166922",
    "e9711749c37b3da33498ec00d78ea41c94d010f2",
    "67f964b1660e5ea92c53dda02fbf454ea5c7afb2",
    "dfff514492c0bca84fa887ec942a1be5044ee67c",
    "7d5ab1c6a59d257d31fde6d4f3da74041024eec4",
    "d76ddce827fdb58d27fbb13f209d7fbc883297aa",
    "66975071179c72e690e62ed4b23b9909a2326f3a",
    "2317e1867996e83267fb985d9622d38712fa78f7",
    "65c978214fbdeba6f10cc2189e96bbb7c3c800ef"
  ]
}