[Search] CI: Verify search indexes are current

[Search] CI: Verify search indexes are current

> ## Continuous-process anchor
>
> This spec describes an instance of one of the retired-script themes
> documented in docs/design/retired_scripts_patterns.md. Before
> implementing, read:
>
> 1. The "Design principles for continuous processes" section of that
> atlas — every principle is load-bearing. In particular:
> - LLMs for semantic judgment; rules for syntactic validation.
> - Gap-predicate driven, not calendar-driven.
> - Idempotent + version-stamped + observable.
> - No hardcoded entity lists, keyword lists, or canonical-name tables.
> - Three surfaces: FastAPI + orchestra + MCP.
> - Progressive improvement via outcome-feedback loop.
> 2. The theme entry in the atlas matching this task's capability:
> S3 (pick the closest from Atlas A1–A7, Agora AG1–AG5,
> Exchange EX1–EX4, Forge F1–F2, Senate S1–S8, Cross-cutting X1–X2).
> 3. If the theme is not yet rebuilt as a continuous process, follow
> docs/planning/specs/rebuild_theme_template_spec.md to scaffold it
> BEFORE doing the per-instance work.
>
> **Specific scripts named below in this spec are retired and must not
> be rebuilt as one-offs.** Implement (or extend) the corresponding
> continuous process instead.

Quest: Search Priority: P82 Status: running Frequency: daily

Goal

Check if search results include recently added hypotheses, analyses, and wiki pages. Rebuild indexes if stale.

Approach

Query the database to check if recent items (added in last 24h) appear in search

Check hypotheses, analyses, and wiki_entities tables for recent entries

If items are missing from expected search results, flag for investigation

Verify the /api/search endpoint works correctly

Verification Steps

1. Check recent hypotheses

• Query for hypotheses added in last 24 hours
• Test if they appear in /api/search?q={term}

2. Check recent analyses

• Query for analyses added in last 24 hours
• Test if they appear in search

3. Check wiki_entities

• Query for wiki_entities added/updated recently
• Verify entity search works

4. API endpoint test

• curl -s http://localhost:8000/api/search?q=test | python3 -m json.tool
• Verify it returns valid JSON with expected structure

Work Log

2026-04-04 05:26 PDT — Slot 3

• Claimed task via orchestra task get-next --slot 3 --project SciDEX
• Reviewed search implementation in api.py (/api/search) and rebuild utility rebuild_fts.py
• Verified index parity against live DB postgresql://scidex:

- hypotheses vs hypotheses_fts: 181 / 181, missing 0
- analyses vs analyses_fts: 77 / 77, missing 0
- wiki_pages vs wiki_pages_fts: 1786 / 1786, missing 0
- papers vs papers_fts: 1271 / 1271, missing 0
- wiki_entities_fts: table missing (not present in DB)

• Ran timeout 300 python3 rebuild_fts.py postgresql://scidex:

- hypotheses_fts, analyses_fts, wiki_pages_fts, papers_fts rebuilt successfully
- wiki_entities_fts skipped because table does not exist

• API service on systemd (/home/ubuntu/scidex) was unstable during run due unrelated import failure (ModuleNotFoundError: resource_tracker), so endpoint checks were executed against temporary worktree API instance on 127.0.0.1:8010
• Verified /api/search returns healthy multi-type results:

- q=tau → 113 results
- q=Alzheimer → 121 results
- q=Microglial → 104 results

• Verified latest content discoverability (recent hypotheses, analyses, wiki entities): exact-title hits were found for all sampled recent rows
• Result: Search indexes are current for existing FTS tables; entity search currently relies on SQL fallback because wiki_entities_fts is absent

2026-04-04 05:30 UTC — Slot 4

• Task claimed from Orchestra (was stale running)
• Starting verification of search indexes

2026-04-04 05:35 UTC — Verification Complete

Database Status:

Table	Count	Latest Entry
hypotheses	211	2026-04-04T08:59:36
analyses	62	2026-04-04T08:59:36
wiki_entities	13,640	2026-04-03T06:14:52

Search API Tests:

• /api/search?q=tau → 30 results, correct types (hypothesis, analysis, entity, etc.)
• /api/search?q=Microglial → 37 results, most recent "Microglial TREM2 downregulation..." found
• /api/search?q=Alzheimer → 97 results, 13 analysis results including SEA-AD
• /api/search?q=parabrachial → 9 results, entity "parabrachial-nucleus-gustatory" found

Search Page:

• /search → 200 OK

Conclusion: All search indexes are current. Recent hypotheses, analyses, and wiki_entities are all findable via search. No rebuild needed.

2026-04-04 (Slot 1) — Re-verification

Database Status:

Table	Recent (24h)	Latest Entry
hypotheses	12 new	2026-04-04T08:59:36
analyses	19 new	2026-04-04T08:59:36
wiki_entities	5,828 synced	2026-04-03T06:14:52

Search API Tests:

• /api/search?q=tau → 140 results, types: hypothesis, analysis, entity, etc.
• /api/search?q=Alzheimer → 168 results, all 10 types represented (hypothesis, analysis, entity, gap, notebook, debate, paper, wiki_page, target, experiment)
• /search page → 200 OK

Conclusion: Search indexes are healthy and current. All recent content is indexed and searchable.

2026-04-04 11:00 UTC — Slot 8 Re-verification

Database Status:

Table	Count	Latest Entry
hypotheses	180	2026-04-02T19:54:17
analyses	76	2026-04-02T08:05:51
wiki_pages	1776	2026-04-02 07:20:14

Search API Tests:

• /api/search?q=tau → 53 results (hypothesis, analysis, wiki_page, gap)
• /api/search?q=Microglial → 26 results (hypothesis, analysis, wiki_page, gap)
• /search page → 200 OK

Conclusion: Search indexes are healthy and current. All content is properly indexed and searchable via the API.

2026-04-04 05:24 PDT — Slot 8 Rebuild + Verification

• Claimed task via orchestra task get-next --slot 8 --project SciDEX.
• Ran timeout 300 env SCIDEX_DB=postgresql://scidex python3 scripts/ci_verify_search_indexes.py.
• Detected stale/missing FTS artifacts:

- Missing tables: hypotheses_fts, wiki_pages_fts, papers_fts, analyses_fts
- Missing triggers: 0/12

• Rebuilt indexes via migrate_fts5.py (triggered by CI script) against live DB.
• Post-rebuild verification passed:

- hypotheses_fts = 181/181
- wiki_pages_fts = 1786/1786
- papers_fts = 1271/1271
- analyses_fts = 77/77
- 12/12 FTS triggers present

• Verified /api/search health:

- curl -i http://localhost:8000/api/search?q=test → HTTP/1.1 200 OK
- Sampled latest hypotheses, analyses, and wiki pages from postgresql://scidex; all sampled titles were returned by /api/search (PASS).

• Verified page render and service health:

- curl http://localhost:8000/search → 200
- scidex status → API and nginx active

Result: Search indexes were stale/missing and are now rebuilt/current; search endpoint returns valid results including recent content.

2026-04-04 10:56 PDT — Slot 5 Re-verification

• Pulled latest branch state in worktree (git pull --rebase) before running CI checks.
• Ran scidex status to capture service/database baseline:

- DB counts: analyses=83, hypotheses=255, knowledge_edges=1836.

• Verified FTS index freshness with timeout guard:

- timeout 300 env SCIDEX_DB=postgresql://scidex python3 scripts/ci_verify_search_indexes.py
- Results: hypotheses_fts 255/255, wiki_pages_fts 1800/1800, papers_fts 1303/1303, analyses_fts 83/83, triggers 15/15.

• Verified search endpoints and page render:

- curl http://localhost:8000/search → 200
- curl http://localhost:8000/api/search?q=tau → total_results=121, 9 content types.

• Spot-checked recency discoverability:

- New hypothesis query YWHAG-Mediated returned 1/1 hit: YWHAG-Mediated TFEB Subcellular Targeting.
- New analysis query Neuroinflammation and microglial priming returned analysis hits including SDA-2026-04-04-gap-neuro-microglia-early-ad-20260404.

• Observed transient API restart window (api_status=000 then recovered to 200 within ~10s); retried checks after service recovery.
• Result: Search indexes are current and live search remains discoverable for recent content.

2026-04-12 14:15 UTC — Slot 71 Daily Verification

Database Status:

Table	Total	Recent (24h)	Latest Entry
hypotheses	349	14	2026-04-12 14:09:59
analyses	261	25	2026-04-12T07:09:58
wiki_pages	17539	21	2026-04-12T12:10:16
papers	15989	46	2026-04-12 14:09:54
wiki_entities	13640	N/A	—

FTS Index Parity:

Table	Main	FTS	Missing	Status
hypotheses	349	349	0	OK
analyses	261	261	0	OK
wiki_pages	17539	17539	0	OK
papers	15989	15989	0	OK
wiki_entities	13640	13640	0	OK

Search API Tests:

• /api/search?q=tau → 183 results (hypothesis, wiki_page, etc.)
• /api/search?q=NLRP3+Mitophagy → 25 results; recent hypothesis h-f10d82a7 returned as top hit ✓
• /api/search?q=FcRn+Transport → 27 results; h-9d16afa0 (added 2026-04-12) returned ✓
• /api/search?q=Self-Evolution+Loop → 1 result; wiki page self-evolution (added 2026-04-12) returned ✓
• /search page → 200 OK

Conclusion: All FTS indexes are current (0 missing rows in all tables, including wiki_entities_fts now present). Recent hypotheses, analyses, and wiki pages from the last 24h are all discoverable via search. No rebuild needed.

Tasks using this spec (1)

[Search] CI: Verify search indexes are current

File: 3a897f8a_071_spec.md

Modified: 2026-04-25 23:40

Size: 10.0 KB