SciDEX — Task: [Forge] Add analysis environment health checks and

Completion Notes

Auto-completed by supervisor after successful deploy to main

Git Commits (2)

Squash merge: orchestra/task/f1e7a44b-add-analysis-environment-health-checks-a (1 commits)2026-04-25

[Forge] Add analysis environment health checks and corruption detection [task:f1e7a44b4b98]2026-04-03

Spec File

[Forge] Add analysis environment health checks and corruption detection

Quest: Analysis Sandboxing Priority: P3 Status: open

Goal

Before and after each analysis, run health checks to verify the system is not corrupted. Check DB integrity, verify key tables, confirm services are running, and flag any unexpected file modifications outside the analysis temp directory.

Acceptance Criteria

☐ Pre-analysis health check: DB integrity, service status, disk space

☐ Post-analysis health check: same checks + diff of unexpected file changes

☐ Health check results stored in analyses table (pre_health_check, post_health_check)

☐ Analysis marked as 'corrupted' if post-check finds unexpected changes

☐ Senate dashboard shows health check pass/fail rates

Approach

Write health_check.py with DB PRAGMA integrity_check, service pings, disk space

Add file watchdog: snapshot /home/ubuntu/scidex/*.py mtimes before analysis

Compare after analysis — flag any changes outside temp dir

Integrate into orchestrator: run_health_check() before and after

Add migration for health check columns on analyses table

Dependencies

_Identify during implementation._

Dependents

_Identify during implementation._

Work Log

2026-04-25 06:30 UTC — Slot minimax:70

Task started: Verified task is still relevant (prior commit ca46c3634 not on main)
Challenge: Prior health_check.py was SQLite-based; PostgreSQL is now the sole datastore
Rebase: Rebased onto current origin/main

Implementation

Created scidex/forge/health_check.py (311 lines):

- check_db_integrity() — PostgreSQL connectivity check
- check_db_tables() — Verify 11 critical tables exist
- check_disk_space() — Monitor free disk space
- check_api_service() — HTTP probe to localhost:8000
- check_nginx_service() — systemctl check for nginx
- snapshot_file_state() — Track .py file mtimes/size/hashes
- compare_snapshots() — Detect unexpected file changes
- run_pre_analysis_health_check() — Pre-analysis health check
- run_post_analysis_health_check() — Post-analysis with comparison
- evaluate_health_status() — Overall status evaluation
- is_analysis_corrupted() — Corruption detection logic
- store_health_check() — Persist results to analyses table

Created migrations/111_add_health_check_columns.py:

- Adds pre_health_check (JSONB)
- Adds post_health_check (JSONB)
- Adds is_corrupted (BOOLEAN DEFAULT FALSE)
- Migration applied successfully

Updated scidex/forge/executor.py LocalExecutor.run_analysis():

- Runs pre-analysis health check before launching cgroup subprocess
- Runs post-analysis health check after completion
- Stores results via store_health_check()
- Marks analysis corrupted via is_corrupted() when appropriate
- Audit logging of health check results

Verification:

- All imports working
- DB integrity: pass
- DB tables: pass (11/11 required tables found)
- Disk space: pass (544.38 GB free)
- API service: pass
- File changes: pass (no unexpected modifications)

Files Changed

scidex/forge/health_check.py (new)
migrations/111_add_health_check_columns.py (new)
scidex/forge/executor.py (modified run_analysis)
docs/planning/specs/f1e7a44b4b98_forge_add_analysis_environment_health_c_spec.md (work log)

Acceptance Criteria Status

☑ Pre-analysis health check: DB integrity, service status, disk space

☑ Post-analysis health check: same checks + diff of unexpected file changes

☑ Health check results stored in analyses table (pre_health_check, post_health_check)

☑ Analysis marked as 'corrupted' if post-check finds unexpected changes

☐ Senate dashboard shows health check pass/fail rates (deferred — requires Senate integration)

Payload JSON

{
  "requirements": {
    "coding": 7,
    "reasoning": 7,
    "analysis": 8
  },
  "completion_shas": [
    "17cbe60b2"
  ],
  "completion_shas_checked_at": ""
}