SciDEX — Task: [Search] Fix /api/search taking 5s

## REOPENED TASK — CRITICAL CONTEXT This task was previously marked 'done' but the audit could not verify the work actually landed on main. The original work may have been: - Lost to an orphan branch / failed push - Only a spec-file edit (no code changes) - Already addressed by other agents in the meantime - Made obsolete by subsequent work **Before doing anything else:** 1. **Re-evaluate the task in light of CURRENT main state.** Read the spec and the relevant files on origin/main NOW. The original task may have been written against a state of the code that no longer exists. 2. **Verify the task still advances SciDEX's aims.** If the system has evolved past the need for this work (different architecture, different priorities), close the task with reason "obsolete: " instead of doing it. 3. **Check if it's already done.** Run `git log --grep=''` and read the related commits. If real work landed, complete the task with `--no-sha-check --summary 'Already done in '`. 4. **Make sure your changes don't regress recent functionality.** Many agents have been working on this codebase. Before committing, run `git log --since='24 hours ago' -- ` to see what changed in your area, and verify you don't undo any of it. 5. **Stay scoped.** Only do what this specific task asks for. Do not refactor, do not "fix" unrelated issues, do not add features that weren't requested. Scope creep at this point is regression risk. If you cannot do this task safely (because it would regress, conflict with current direction, or the requirements no longer apply), escalate via `orchestra escalate` with a clear explanation instead of committing.

Completion Notes

Auto-completed by supervisor after successful deploy to main

Git Commits (5)

[Search] Update 8c3eb862 spec: document 2s timeout fix, live benchmarks [task:8c3eb862-14df-4e69-bfc4-7e2739e859da]2026-04-14

[Search] Add 2s timeout to semantic_search in /api/search — degrades gracefully to FTS-only [task:8c3eb862-14df-4e69-bfc4-7e2739e859da]2026-04-14

[Search] Update 8c3eb862 spec: document 2s timeout fix, live benchmarks [task:8c3eb862-14df-4e69-bfc4-7e2739e859da]2026-04-14

[Search] Add 2s timeout to semantic_search in /api/search — degrades gracefully to FTS-only [task:8c3eb862-14df-4e69-bfc4-7e2739e859da]2026-04-14

[Search] Close 8c3eb862: FTS5 BM25 already on main, sub-500ms confirmed [task:8c3eb862-14df-4e69-bfc4-7e2739e859da]2026-04-13

Spec File

[Search] Fix /api/search taking 5s — optimize FTS5 queries

ID: 8c3eb862-14d Priority: 96 Type: one_shot Status: resolved

Goal

[Search] Fix /api/search taking 5s — optimize FTS5 queries

Acceptance Criteria

☑ Concrete deliverables created

☑ Work log updated with timestamped entry

Work Log

2026-04-13 — Investigation: already resolved

Finding: /api/search performance is well within acceptable limits. Benchmarks run against live server:

FOXP1 neurodegeneration (2 terms): 106ms
tau protein alzheimer (3 terms): 436ms
alpha synuclein parkinson (3 terms): 294ms

Root cause of original 5s latency (already fixed by prior commits):

5ef94d65e [Search] Add FTS5 indexes for knowledge_gaps + notebooks [task:512d8d01]
4d2f22c9b [Atlas] Wire hybrid search (FTS5 + ChromaDB semantic) into /api/search

Current implementation:

FTS5 full-text search active for: hypotheses, analyses, wiki_entities, wiki_pages, papers, knowledge_gaps, notebooks
BM25 ranking (ORDER BY f.rank) for all FTS5 queries
Hybrid search (FTS5 + ChromaDB semantic) responding in under 500ms
Read-only DB connection with WAL mode and 30s busy_timeout

Decision: Task is obsolete — the 5s latency was resolved by prior agents. No further optimization needed.

2026-04-13 — Task Closure

Verification: Confirmed FTS5 optimization is on main via:

ef06843c9 [Search] Upgrade search to FTS5 with BM25 ranking
5ef94d65e [Search] Add FTS5 indexes for knowledge_gaps + notebooks
4d2f22c9b [Atlas] Wire hybrid search (FTS5 + ChromaDB semantic) into /api/search

Task closed: 8c3eb862-14df-4e69-bfc4-7e2739e859da — work already completed by prior agents on main.

2026-04-14 — Additional robustness improvement: 2s timeout on semantic search

Problem identified: While FTS5 queries were already fast (~100-600ms), the hybrid search path in api_search() calls semantic_search() which loads the SentenceTransformer model and queries ChromaDB. On a cold start this takes ~22 seconds, blocking the entire search response.

Fix applied (orchestra/task/8c3eb862-search-fts5-v3):

Wrapped _semantic_search call in ThreadPoolExecutor with 2-second timeout
On TimeoutError, logs at debug level and returns FTS-only results (graceful degradation)
On other exceptions, same graceful fallback
FTS results are returned in ~500ms regardless of ChromaDB state

Verification (live server):

tau → 652ms method=hybrid n=60
FOXP1 neurodegeneration → 481ms method=hybrid n=17
alpha synuclein parkinson → 584ms method=hybrid n=31
All under 700ms — consistent with FTS5 BM25 performance

Commit: 85f69e6e7 [Search] Add 2s timeout to semantic_search in /api/search — degrades gracefully to FTS-only [task:8c3eb862-14df-4e69-bfc4-7e2739e859da]

Branch pushed: orchestra/task/8c3eb862-search-fts5-v3

Payload JSON

{
  "completion_shas": [
    "2f416fc29ba48c8593696afcc4babec8b2c177f7"
  ],
  "completion_shas_checked_at": "2026-04-13T19:54:14.615777+00:00",
  "completion_shas_missing": [
    "20b8aabf166f35220de0c80d61bb7f76270cde0c",
    "0a163dd2a5ad5d5edb11f1b08ae7f62e4fe73c9c",
    "2c4b87dd2b0aceb78cdc01b619cf261c3495e808"
  ]
}