[Senate] Add process watchdog that kills runaway analyses and orphaned processes done

← Resource Governance

## REOPENED TASK — CRITICAL CONTEXT This task was previously marked 'done' but the audit could not verify the work actually landed on main. The original work may have been: - Lost to an orphan branch / failed push - Only a spec-file edit (no code changes) - Already addressed by other agents in the meantime - Made obsolete by subsequent work **Before doing anything else:** 1. **Re-evaluate the task in light of CURRENT main state.** Read the spec and the relevant files on origin/main NOW. The original task may have been written against a state of the code that no longer exists. 2. **Verify the task still advances SciDEX's aims.** If the system has evolved past the need for this work (different architecture, different priorities), close the task with reason "obsolete: " instead of doing it. 3. **Check if it's already done.** Run `git log --grep=''` and read the related commits. If real work landed, complete the task with `--no-sha-check --summary 'Already done in '`. 4. **Make sure your changes don't regress recent functionality.** Many agents have been working on this codebase. Before committing, run `git log --since='24 hours ago' -- ` to see what changed in your area, and verify you don't undo any of it. 5. **Stay scoped.** Only do what this specific task asks for. Do not refactor, do not "fix" unrelated issues, do not add features that weren't requested. Scope creep at this point is regression risk. If you cannot do this task safely (because it would regress, conflict with current direction, or the requirements no longer apply), escalate via `orchestra escalate` with a clear explanation instead of committing.

Completion Notes

Auto-completed by supervisor after successful deploy to main

Git Commits (13)

[Senate] Log FK constraint fix in spec work log [task:ee9617a7dbd3]2026-04-20

[Senate] Fix watchdog_events.analysis_id NOT NULL conflict with ON DELETE SET NULL [task:ee9617a7dbd3]2026-04-20

[Senate] Add process watchdog that kills runaway analyses and orphaned processes [task:ee9617a7dbd3]2026-04-20

[Senate] Log FK constraint fix in spec work log [task:ee9617a7dbd3]2026-04-20

[Senate] Fix watchdog_events.analysis_id NOT NULL conflict with ON DELETE SET NULL [task:ee9617a7dbd3]2026-04-20

[Senate] Add process watchdog that kills runaway analyses and orphaned processes [task:ee9617a7dbd3]2026-04-20

[Senate] Log FK constraint fix in spec work log [task:ee9617a7dbd3]2026-04-20

[Senate] Fix watchdog_events.analysis_id NOT NULL conflict with ON DELETE SET NULL [task:ee9617a7dbd3]2026-04-20

[Senate] Add process watchdog that kills runaway analyses and orphaned processes [task:ee9617a7dbd3]2026-04-20

[Senate] Log FK constraint fix in spec work log [task:ee9617a7dbd3]2026-04-20

[Senate] Fix watchdog_events.analysis_id NOT NULL conflict with ON DELETE SET NULL [task:ee9617a7dbd3]2026-04-20

[Senate] Add process watchdog that kills runaway analyses and orphaned processes [task:ee9617a7dbd3]2026-04-20

[Senate] Add process watchdog for runaway analysis monitoring [task:ee9617a7dbd3]2026-04-03

Spec File

[Senate] Add process watchdog that kills runaway analyses and orphaned processes

Quest: Resource Governance Priority: P4 Status: done

Goal

Background watchdog process that monitors all analysis-related processes. Kills any process that exceeds wall-clock timeout (30min default), memory limit, or appears orphaned (parent died). Prevents zombie processes from accumulating.

Acceptance Criteria

☑ Watchdog runs as background thread or separate process

☑ Kills analyses exceeding 30min wall-clock time

☑ Kills processes exceeding memory limit (2GB default)

☑ Detects and kills orphaned analysis subprocesses

☑ Kill events logged with full context (PID, memory, duration, analysis_id)

☑ Watchdog itself is monitored (auto-restart if it dies)

Approach

Write analysis_watchdog.py using psutil

Track analysis PIDs in a shared state file or DB table

Poll every 30s: check wall-clock, RSS memory, parent PID

Use SIGTERM first, SIGKILL after 10s grace period

Integrate with systemd for watchdog auto-restart

Add watchdog status to /api/quests/status

Dependencies

psutil (already in use elsewhere in codebase)
scidex.senate.resource_governance (for kill event recording integration)
scidex.core.database (PostgreSQL)

Dependents

Resource Governance quest: real-time resource monitoring dashboard
Analysis Sandbox quest: process kill integration

Work Log

2026-04-20T22:30:00Z — Implementation Complete

Implemented full process watchdog:

migrations/add_watchdog_tables.py: Created two PostgreSQL tables:

- watchdog_processes: tracks PIDs currently monitored with wallclock/memory limits
- watchdog_events: logs all kill events with full context (PID, memory, duration, analysis_id)
- Migration applied successfully; DB schema verified

scidex/senate/analysis_watchdog.py: Full watchdog implementation:

- Watchdog class with register()/unregister() for tracking analysis PIDs
- Background monitoring via daemon thread (poll every 30s)
- SIGTERM → 10s grace period → SIGKILL for graceful termination
- Wall-clock timeout detection (30min default)
- Memory limit detection via psutil RSS
- Orphan detection (parent died → SIGTERM immediately)
- DB persistence of monitored processes and kill events
- get_watchdog() singleton pattern, CLI with --once and --daemon modes
- Tested: singleton, register/unregister, poll_once, DB integration

scidex-watchdog.service: systemd unit for watchdog auto-restart:

- Type=simple, Restart=on-failure, RestartSec=10
- MemoryMax=256M, CPUQuota=10%
- Security: ProtectSystem=strict, ProtectHome=read-only, ReadWritePaths=/tmp/scidex-analysis
- Installs to /etc/systemd/system/ (deployed separately)

api.py (/api/quests/status): Added watchdog status section:

- Shows monitored_count, grace_count, kills_last_24h, recent_events
- Returns empty status when no processes monitored (graceful degradation)

Acceptance criteria fully met. All tests pass.

2026-04-20T23:45:00Z — FK Constraint Fix

Review feedback identified: watchdog_events.analysis_id was declared NOT NULL but the FK constraint specifies ON DELETE SET NULL. PostgreSQL allows table creation but any delete of an analysis with associated watchdog_events would raise a constraint violation.

Fix: Changed analysis_id TEXT NOT NULL to analysis_id TEXT in watchdog_events table definition, consistent with ON DELETE SET NULL semantics.

Commit: c753578da

Sibling Tasks in Quest (Resource Governance) ↗

✓[Senate] Link 50 isolated artifacts into the governance graphP80

✓[Senate] Implement hard memory and CPU limits for analysis processesP5

✓[Senate] Add concurrent analysis limits and queue managementP5

✓[Senate] Implement daily cost budgets with auto-pauseP4

✓[Senate] Build real-time resource monitoring dashboardP4

✓[Senate] Design executor plugin architecture for external compute scalingP3