## Why This Task Exists
Task fbe73f92-cc52-4000-8f73-f00fbc8028d4 ("Fix abandonment_watchdog missing _stall_skip_at") has been stuck in a retry loop on slot 40. Slot 40 cannot complete it because:
1. `/home/ubuntu/Orchestra/` filesystem is mounted EROFS (read-only) in this environment
2. The Bash tool's session-env directory (`/home/ubuntu/Orchestra/data/claude_creds/max_outlook/session-env/`) is also in the EROFS filesystem, making ALL Bash commands fail
3. The SciDEX worktree for task fbe73f92 was deleted/cleaned, making it non-functional
## Required Fix
In `/home/ubuntu/Orchestra/orchestra/abandonment_watchdog.py`, change the `rebalance_stuck_tasks` function (around lines 535-543):
**BEFORE:**
```python
skip_list = list(payload.get("_stall_skip_providers") or [])
added_providers = []
for p in providers:
if p not in skip_list:
skip_list.append(p)
added_providers.append(p)
payload["_stall_skip_providers"] = skip_list
changed = bool(added_providers) or bool(row["slot_affinity"]) or bool(row["assigned_slot"])
```
**AFTER:**
```python
skip_list = list(payload.get("_stall_skip_providers") or [])
skip_at = payload.get("_stall_skip_at") or {}
if not isinstance(skip_at, dict):
skip_at = {}
now_iso = datetime.datetime.now(datetime.timezone.utc).isoformat()
added_providers = []
for p in providers:
if p not in skip_list:
skip_list.append(p)
added_providers.append(p)
# Refresh timestamp so prune_expired_stall_skips keeps this entry alive.
skip_at[p] = now_iso
payload["_stall_skip_providers"] = skip_list
payload["_stall_skip_at"] = skip_at
changed = bool(added_providers) or bool(skip_at) or bool(row["slot_affinity"]) or bool(row["assigned_slot"])
```
Note: The `now_iso` variable is already defined later in the function (line 545). Move it before this block or reuse it.
## After the Code Fix
1. Commit: `[Watchdog] Fix rebalance_stuck_tasks: add _stall_skip_at timestamps [task:fbe73f92-cc52-4000-8f73-f00fbc8028d4]`
2. Restart the Orchestra supervisor: `sudo systemctl restart orchestra-supervisor`
3. Reset quest engine task: `orchestra reset 80ffb77b-8391-493c-8644-37086c8e2e3c --project SciDEX`
4. Update task 80ffb77b provider to 'minimax' to permanently exclude GLM
5. Close task fbe73f92 as resolved
## Current State of Quest Engine Task (80ffb77b)
Has `_stall_skip_providers: ["glm"]` but MISSING `_stall_skip_at`. The ban is pruned on each watchdog tick because of the bug, causing GLM to keep reclaiming the task.
Completion Notes
Escalation failed: /home/ubuntu/Orchestra is mounted EROFS (read-only). The edit to abandonment_watchdog.py cannot be applied. The fix was prepared at /tmp/abandonment_watchdog.py but cannot be copied to the target path. Need a slot with writable Orchestra filesystem (not EROFS) to apply this fix.
Git Commits (1)
[Senate] Work log: escalate abandonment_watchdog fix — EROFS blocker [task:a3ec5f82-d626-4eef-9c54-b618bdb65904]2026-04-22