[Forge] Add analysis environment health checks and corruption detection
Quest: Analysis Sandboxing
Priority: P3
Status: open
Goal
Before and after each analysis, run health checks to verify the system is not corrupted. Check DB integrity, verify key tables, confirm services are running, and flag any unexpected file modifications outside the analysis temp directory.
Acceptance Criteria
☐ Pre-analysis health check: DB integrity, service status, disk space
☐ Post-analysis health check: same checks + diff of unexpected file changes
☐ Health check results stored in analyses table (pre_health_check, post_health_check)
☐ Analysis marked as 'corrupted' if post-check finds unexpected changes
☐ Senate dashboard shows health check pass/fail rates
Approach
Write health_check.py with DB PRAGMA integrity_check, service pings, disk space
Add file watchdog: snapshot /home/ubuntu/scidex/*.py mtimes before analysis
Compare after analysis — flag any changes outside temp dir
Integrate into orchestrator: run_health_check() before and after
Add migration for health check columns on analyses tableDependencies
_Identify during implementation._
Dependents
_Identify during implementation._
Work Log
2026-04-25 06:30 UTC — Slot minimax:70
- Task started: Verified task is still relevant (prior commit ca46c3634 not on main)
- Challenge: Prior health_check.py was SQLite-based; PostgreSQL is now the sole datastore
- Rebase: Rebased onto current origin/main
Implementation
Created scidex/forge/health_check.py (311 lines):
-
check_db_integrity() — PostgreSQL connectivity check
-
check_db_tables() — Verify 11 critical tables exist
-
check_disk_space() — Monitor free disk space
-
check_api_service() — HTTP probe to localhost:8000
-
check_nginx_service() — systemctl check for nginx
-
snapshot_file_state() — Track
.py file mtimes/size/hashes
-
compare_snapshots() — Detect unexpected file changes
-
run_pre_analysis_health_check() — Pre-analysis health check
-
run_post_analysis_health_check() — Post-analysis with comparison
-
evaluate_health_status() — Overall status evaluation
-
is_analysis_corrupted() — Corruption detection logic
-
store_health_check() — Persist results to analyses table
Created migrations/111_add_health_check_columns.py:
- Adds
pre_health_check (JSONB)
- Adds
post_health_check (JSONB)
- Adds
is_corrupted (BOOLEAN DEFAULT FALSE)
- Migration applied successfully
Updated scidex/forge/executor.py LocalExecutor.run_analysis():
- Runs pre-analysis health check before launching cgroup subprocess
- Runs post-analysis health check after completion
- Stores results via
store_health_check() - Marks analysis corrupted via
is_corrupted() when appropriate
- Audit logging of health check results
Verification:
- All imports working
- DB integrity: pass
- DB tables: pass (11/11 required tables found)
- Disk space: pass (544.38 GB free)
- API service: pass
- File changes: pass (no unexpected modifications)
Files Changed
scidex/forge/health_check.py (new)migrations/111_add_health_check_columns.py (new)scidex/forge/executor.py (modified run_analysis)docs/planning/specs/f1e7a44b4b98_forge_add_analysis_environment_health_c_spec.md (work log)
Acceptance Criteria Status
☑ Pre-analysis health check: DB integrity, service status, disk space
☑ Post-analysis health check: same checks + diff of unexpected file changes
☑ Health check results stored in analyses table (pre_health_check, post_health_check)
☑ Analysis marked as 'corrupted' if post-check finds unexpected changes
☐ Senate dashboard shows health check pass/fail rates (deferred — requires Senate integration)