[Forge] Design pluggable executor interface for future container/cloud migration done coding:8 safety:9

← Analysis Sandboxing
## REOPENED TASK — CRITICAL CONTEXT This task was previously marked 'done' but the audit could not verify the work actually landed on main. The original work may have been: - Lost to an orphan branch / failed push - Only a spec-file edit (no code changes) - Already addressed by other agents in the meantime - Made obsolete by subsequent work **Before doing anything else:** 1. **Re-evaluate the task in light of CURRENT main state.** Read the spec and the relevant files on origin/main NOW. The original task may have been written against a state of the code that no longer exists. 2. **Verify the task still advances SciDEX's aims.** If the system has evolved past the need for this work (different architecture, different priorities), close the task with reason "obsolete: " instead of doing it. 3. **Check if it's already done.** Run `git log --grep=''` and read the related commits. If real work landed, complete the task with `--no-sha-check --summary 'Already done in '`. 4. **Make sure your changes don't regress recent functionality.** Many agents have been working on this codebase. Before committing, run `git log --since='24 hours ago' -- ` to see what changed in your area, and verify you don't undo any of it. 5. **Stay scoped.** Only do what this specific task asks for. Do not refactor, do not "fix" unrelated issues, do not add features that weren't requested. Scope creep at this point is regression risk. If you cannot do this task safely (because it would regress, conflict with current direction, or the requirements no longer apply), escalate via `orchestra escalate` with a clear explanation instead of committing.

Completion Notes

Auto-completed by supervisor after successful deploy to main

Git Commits (5)

Squash merge: orchestra/task/603329eb-design-pluggable-executor-interface-for (1 commits)2026-04-20
Squash merge: orchestra/task/603329eb-design-pluggable-executor-interface-for (1 commits)2026-04-20
Squash merge: orchestra/task/603329eb-design-pluggable-executor-interface-for (1 commits)2026-04-20
Squash merge: orchestra/task/603329eb-design-pluggable-executor-interface-for (1 commits)2026-04-20
[Forge] Design pluggable executor interface for container/cloud migration2026-04-13
Spec File

[Forge] Design pluggable executor interface for future container/cloud migration

Quest: Analysis Sandboxing Priority: P4 Status: completed

Goal

Create an abstract Executor interface that the orchestrator uses to run analyses. Implement LocalExecutor (cgroup-based, current) and stub out ContainerExecutor (Docker) and CloudExecutor (AWS Batch). This ensures the sandboxing design supports future scaling without rewriting the orchestrator.

The interface should also cover the intermediate step SciDEX needs now:
explicit named local runtimes backed by existing conda environments, with
relaunch, tempdir isolation, and execution logging. Container and cloud
executors should extend that contract rather than bypass it.

Acceptance Criteria

☑ Abstract Executor base class with run_analysis(), check_status(), kill(), get_logs() methods
☑ LocalExecutor implements cgroup-based execution (current VM)
☑ Local named-runtime execution is supported and leaves auditable logs
☑ ContainerExecutor stub with Docker run command structure
☑ CloudExecutor stub with AWS Batch job submission structure
☑ Orchestrator uses Executor interface, switchable via config
☑ Design doc in spec explaining the executor architecture

Approach

  • Define Executor ABC in executor.py
  • Implement LocalExecutor wrapping the cgroup sandbox
  • Stub ContainerExecutor with Docker SDK patterns
  • Stub CloudExecutor with boto3 Batch patterns
  • Add EXECUTOR_TYPE config (local/container/cloud) to .env
  • Refactor scidex_orchestrator.py to use the executor interface
  • Dependencies

    • scidex/senate/cgroup_isolation.py — cgroup-based isolation (Senate layer)
    • docker Python package (for ContainerExecutor)
    • boto3 AWS SDK (for CloudExecutor)

    Dependents

    • scidex_orchestrator.py — uses Executor interface for analysis execution
    • Future container/cloud migration work will implement ContainerExecutor/CloudExecutor

    Design Doc: Executor Architecture

    Overview

    The Executor interface provides a pluggable abstraction for running scientific analyses with isolation and resource controls. It allows the orchestrator to switch between execution backends (local cgroup, Docker containers, AWS Batch) without changing analysis logic.

    Interface Contract

    class Executor(ABC):
        def run_analysis(self, spec: AnalysisSpec) -> str:
            """Launch analysis, return run_id."""
            ...
    
        def check_status(self, run_id: str) -> RunStatus:
            """Check status of a run."""
            ...
    
        def kill(self, run_id: str) -> bool:
            """Terminate a running analysis."""
            ...
    
        def get_logs(self, run_id: str) -> str:
            """Retrieve execution logs."""
            ...

    Execution Flow

  • Specification: AnalysisSpec defines the analysis (command, runtime, resource limits, timeout)
  • Submission: run_analysis() launches the analysis and returns a run_id
  • Monitoring: check_status() polls for completion
  • Retrieval: get_logs() fetches stdout/stderr
  • Control: kill() terminates runaway analyses
  • Executor Types

    TypeImplementationUse Case
    LOCALLocalExecutor wrapping cgroup_isolationCurrent VM execution with cgroup limits
    CONTAINERContainerExecutor with Docker SDKIsolated containerized execution
    CLOUDCloudExecutor with boto3 BatchAWS Batch managed execution

    Named Runtimes

    All executors support named runtimes backed by conda environments:

    • python3 — default Python runtime
    • r-base — R statistical computing
    • julia — Julia numerical computing

    Runtime selection is passed via AnalysisSpec.runtime and the executor maps it to the appropriate environment activation.

    Resource Limits

    AnalysisSpec defines resource constraints:

    • memory_limit_mb — max RAM in MB
    • cpu_quota — CPU time quota (fraction of core)
    • pids_limit — max number of processes
    • timeout_seconds — wall-clock timeout

    Audit Logging

    LocalExecutor writes execution logs to log_path (if provided) with:

    • Timestamps for start/completion/kill
    • Return codes and exit status
    • Resource usage (memory, CPU)
    • stdout/stderr capture

    Configuration

    Set via EXECUTOR_TYPE environment variable:

    EXECUTOR_TYPE=local      # Default, cgroup-based
    EXECUTOR_TYPE=container   # Docker-based (requires docker package)
    EXECUTOR_TYPE=cloud      # AWS Batch (requires boto3)

    Future Migration Path

  • Phase 1 (Current): LocalExecutor wraps cgroup_isolation for current VM
  • Phase 2 (Container): Deploy ContainerExecutor with Docker images for full isolation
  • Phase 3 (Cloud): Migrate to AWS Batch for elastic scaling
  • The pluggable interface ensures no changes to orchestrator logic when migrating between phases.

    Work Log

    • 2026-04-13: Implemented scidex/forge/executor.py with:
    - Abstract Executor base class with run_analysis(), check_status(), kill(), get_logs() methods
    - AnalysisSpec dataclass for passing run parameters
    - RunResult dataclass for returning results
    - RunStatus enum (PENDING, RUNNING, COMPLETED, FAILED, KILLED, UNKNOWN)
    - ExecutorType enum (LOCAL, CONTAINER, CLOUD)
    - LocalExecutor wrapping cgroup_isolation.isolated_run(), with named runtime (conda env) support and audit logging
    - ContainerExecutor stub using Docker SDK patterns (requires docker package)
    - CloudExecutor stub using boto3 Batch patterns (requires boto3 package)
    - get_executor() factory function reading EXECUTOR_TYPE from env
    - Created .env.example documenting configuration options
    - Note: Orchestrator integration and full design doc still pending (separate task)

    • 2026-04-13: Added full design doc section explaining executor architecture:
    - Interface contract and execution flow
    - Executor types table (LOCAL/CONTAINER/CLOUD)
    - Named runtimes support
    - Resource limits specification
    - Audit logging design
    - Configuration via EXECUTOR_TYPE env var
    - Future migration path (Phase 1→2→3)
    - Marked spec as completed
    - Task ID referenced: a4f7702f-0e88-41ca-8327-88d9e564ba02

    • 2026-04-20: Re-verified work is present on origin/main (blob 6eca5773394832):
    - scidex/forge/executor.py contains full implementation: Executor ABC, LocalExecutor, ContainerExecutor, CloudExecutor, get_executor() factory
    - scidex/agora/scidex_orchestrator.py imports and uses Executor interface (lines 82-88, 554-562, 661-694)
    - No commit with task ID 603329ebdcb3 exists (original work done under a4f7702f)
    - Orchestrator uses get_executor() with EXECUTOR_TYPE env var for pluggable backends
    - Conclusion: Work is complete; completing task with --no-sha-check
    - Evidence: git ls-tree origin/main scidex/forge/executor.py → blob 6eca5773394832d7a6022752594bddfbedcf2173

    Payload JSON
    {
      "requirements": {
        "coding": 8,
        "safety": 9
      }
    }

    Sibling Tasks in Quest (Analysis Sandboxing) ↗