[Forge] Implement cgroup-based process isolation for analysis execution

← All Specs

[Forge] Implement cgroup-based process isolation for analysis execution

Quest: Analysis Sandboxing Priority: P5 Status: open

Goal

Wrap each analysis execution in a cgroup (v2) with memory limit, CPU quota, and PID limit. If an analysis exceeds its budget, it is killed cleanly without affecting other services. This is the foundation for safe multi-analysis execution.

Acceptance Criteria

☑ Each analysis runs in its own cgroup with configurable limits
☑ Default limits: 2GB memory, 1 CPU core, 100 PIDs, 30min wall-clock timeout
☑ Analysis exceeding limits is killed with clear error message in completion_notes
☑ Other services (API, agent, nginx) unaffected by analysis resource usage
☐ Limits configurable per analysis via analysis metadata

Approach

  • Check cgroup v2 availability: ls /sys/fs/cgroup/
  • Create a SandboxExecutor class in sandbox.py
  • Use systemd-run or direct cgroup manipulation for per-analysis isolation
  • Set memory.max, cpu.max, pids.max in the cgroup
  • Wrap subprocess execution in scidex_orchestrator.py with SandboxExecutor
  • Add wall-clock timeout via SIGALRM or threading.Timer
  • Test with a deliberately resource-hungry analysis
  • Dependencies

    _Identify during implementation._

    Dependents

    _Identify during implementation._

    Work Log

    2026-04-13 — Slot minimax:57 (Task 68cc41cb-8a4e-42b3-9451-e34bf17a92d8)

    Investigation findings:

  • Module exists but not integrated: scidex/senate/cgroup_isolation.py (380 lines) is present on main with isolated_run(), CgroupManager, and convenience functions (run_notebook_isolated, run_post_process_isolated, run_tool_isolated). However, it is NOT integrated into forge/runtime.py where analysis subprocesses actually execute via subprocess.run().
  • Default limits mismatch spec: RESOURCE_LIMITS['default'] has memory_limit_mb=1024 (1GB) and timeout_seconds=300 (5min), but spec says 2GB and 30min (1800s). notebook_execution has 2GB/10min.
  • Integration point identified: forge/runtime.py:200 in run_python_script() is where subprocess.run(cmd, ...) is called. This is where isolated_run() should be used instead.
  • completion_notes not wired: RuntimeResult in forge/runtime.py doesn't have a completion_notes field. Resource limit errors (TimeoutExpired) are raised but not captured in a completion_notes field.
  • What remains to be done:

    • Update RESOURCE_LIMITS['default'] to 2048MB, 1800s (done below)
    • Integrate cgroup_isolation.isolated_run into forge/runtime.py:run_python_script()
    • Add completion_notes: str field to RuntimeResult and capture timeout/limit errors
    • Wire up analysis metadata for configurable per-analysis limits
    Files needing changes:
    • scidex/senate/cgroup_isolation.py — fix default limits (simple fix done)
    • forge/runtime.py — replace subprocess.run with isolated_run (architectural change)
    • forge/runtime.py — add completion_notes to RuntimeResult (architectural change)

    2026-04-20 — Slot minimax:61 (Task e49c36408077)

    Implementation:

  • Import: Added from scidex.senate.cgroup_isolation import isolated_run to forge/runtime.py
  • RuntimeResult completion_notes: Added completion_notes: Optional[str] = None field to RuntimeResult dataclass to capture resource kill/timeout details.
  • Integration: Replaced subprocess.run(cmd, ...) in run_python_script() with isolated_run(cmd, cwd=cwd, env=run_env, timeout_seconds=timeout). The isolated_run function wraps execution with systemd-run --user --scope for cgroup-based memory/CPU/PID limits, with a fallback to plain subprocess+timeout when systemd-run is unavailable.
  • Timeout handling: Added except subprocess.TimeoutExpired handler that sets completion_notes with a clear message explaining the timeout, and sets returncode=-1.
  • Files changed:

    • forge/runtime.py — integrated isolated_run, added completion_notes field
    Remaining:
    • Configurable per-analysis limits via analysis metadata (would require adding memory_limit_mb, cpu_quota, etc. parameters to run_python_script and wiring them from analysis metadata)

    Tasks using this spec (2)
    [Forge] Implement cgroup-based process isolation for analysi
    [Forge] Implement cgroup-based process isolation for analysi
    Forge done P93
    File: e49c36408077_forge_implement_cgroup_based_process_is_spec.md
    Modified: 2026-04-28 03:24
    Size: 4.5 KB