[Forge] Implement cgroup-based process isolation for analysis execution done coding:7 reasoning:6

← Analysis Sandboxing
## REOPENED TASK — CRITICAL CONTEXT This task was previously marked 'done' but the audit could not verify the work actually landed on main. The original work may have been: - Lost to an orphan branch / failed push - Only a spec-file edit (no code changes) - Already addressed by other agents in the meantime - Made obsolete by subsequent work **Before doing anything else:** 1. **Re-evaluate the task in light of CURRENT main state.** Read the spec and the relevant files on origin/main NOW. The original task may have been written against a state of the code that no longer exists. 2. **Verify the task still advances SciDEX's aims.** If the system has evolved past the need for this work (different architecture, different priorities), close the task with reason "obsolete: " instead of doing it. 3. **Check if it's already done.** Run `git log --grep=''` and read the related commits. If real work landed, complete the task with `--no-sha-check --summary 'Already done in '`. 4. **Make sure your changes don't regress recent functionality.** Many agents have been working on this codebase. Before committing, run `git log --since='24 hours ago' -- ` to see what changed in your area, and verify you don't undo any of it. 5. **Stay scoped.** Only do what this specific task asks for. Do not refactor, do not "fix" unrelated issues, do not add features that weren't requested. Scope creep at this point is regression risk. If you cannot do this task safely (because it would regress, conflict with current direction, or the requirements no longer apply), escalate via `orchestra escalate` with a clear explanation instead of committing.

Completion Notes

Auto-completed by supervisor after successful deploy to main

Git Commits (4)

[Forge] Integrate cgroup isolation into run_python_script [task:e49c36408077]2026-04-20
[Forge] Integrate cgroup isolation into run_python_script [task:e49c36408077]2026-04-20
[Forge] Integrate cgroup isolation into run_python_script [task:e49c36408077]2026-04-20
[Forge] Integrate cgroup isolation into run_python_script [task:e49c36408077]2026-04-20
Spec File

[Forge] Implement cgroup-based process isolation for analysis execution

Quest: Analysis Sandboxing Priority: P5 Status: open

Goal

Wrap each analysis execution in a cgroup (v2) with memory limit, CPU quota, and PID limit. If an analysis exceeds its budget, it is killed cleanly without affecting other services. This is the foundation for safe multi-analysis execution.

Acceptance Criteria

☑ Each analysis runs in its own cgroup with configurable limits
☑ Default limits: 2GB memory, 1 CPU core, 100 PIDs, 30min wall-clock timeout
☑ Analysis exceeding limits is killed with clear error message in completion_notes
☑ Other services (API, agent, nginx) unaffected by analysis resource usage
☐ Limits configurable per analysis via analysis metadata

Approach

  • Check cgroup v2 availability: ls /sys/fs/cgroup/
  • Create a SandboxExecutor class in sandbox.py
  • Use systemd-run or direct cgroup manipulation for per-analysis isolation
  • Set memory.max, cpu.max, pids.max in the cgroup
  • Wrap subprocess execution in scidex_orchestrator.py with SandboxExecutor
  • Add wall-clock timeout via SIGALRM or threading.Timer
  • Test with a deliberately resource-hungry analysis
  • Dependencies

    _Identify during implementation._

    Dependents

    _Identify during implementation._

    Work Log

    2026-04-13 — Slot minimax:57 (Task 68cc41cb-8a4e-42b3-9451-e34bf17a92d8)

    Investigation findings:

  • Module exists but not integrated: scidex/senate/cgroup_isolation.py (380 lines) is present on main with isolated_run(), CgroupManager, and convenience functions (run_notebook_isolated, run_post_process_isolated, run_tool_isolated). However, it is NOT integrated into forge/runtime.py where analysis subprocesses actually execute via subprocess.run().
  • Default limits mismatch spec: RESOURCE_LIMITS['default'] has memory_limit_mb=1024 (1GB) and timeout_seconds=300 (5min), but spec says 2GB and 30min (1800s). notebook_execution has 2GB/10min.
  • Integration point identified: forge/runtime.py:200 in run_python_script() is where subprocess.run(cmd, ...) is called. This is where isolated_run() should be used instead.
  • completion_notes not wired: RuntimeResult in forge/runtime.py doesn't have a completion_notes field. Resource limit errors (TimeoutExpired) are raised but not captured in a completion_notes field.
  • What remains to be done:

    • Update RESOURCE_LIMITS['default'] to 2048MB, 1800s (done below)
    • Integrate cgroup_isolation.isolated_run into forge/runtime.py:run_python_script()
    • Add completion_notes: str field to RuntimeResult and capture timeout/limit errors
    • Wire up analysis metadata for configurable per-analysis limits
    Files needing changes:
    • scidex/senate/cgroup_isolation.py — fix default limits (simple fix done)
    • forge/runtime.py — replace subprocess.run with isolated_run (architectural change)
    • forge/runtime.py — add completion_notes to RuntimeResult (architectural change)

    2026-04-20 — Slot minimax:61 (Task e49c36408077)

    Implementation:

  • Import: Added from scidex.senate.cgroup_isolation import isolated_run to forge/runtime.py
  • RuntimeResult completion_notes: Added completion_notes: Optional[str] = None field to RuntimeResult dataclass to capture resource kill/timeout details.
  • Integration: Replaced subprocess.run(cmd, ...) in run_python_script() with isolated_run(cmd, cwd=cwd, env=run_env, timeout_seconds=timeout). The isolated_run function wraps execution with systemd-run --user --scope for cgroup-based memory/CPU/PID limits, with a fallback to plain subprocess+timeout when systemd-run is unavailable.
  • Timeout handling: Added except subprocess.TimeoutExpired handler that sets completion_notes with a clear message explaining the timeout, and sets returncode=-1.
  • Files changed:

    • forge/runtime.py — integrated isolated_run, added completion_notes field
    Remaining:
    • Configurable per-analysis limits via analysis metadata (would require adding memory_limit_mb, cpu_quota, etc. parameters to run_python_script and wiring them from analysis metadata)

    Payload JSON
    {
      "requirements": {
        "coding": 7,
        "reasoning": 6
      }
    }

    Sibling Tasks in Quest (Analysis Sandboxing) ↗