Scheduled dedup scanning done

← Artifact Lifecycle

Add daily cron job running artifact_dedup_agent.run_full_scan() to continuously detect sprawl. ## REOPENED TASK — CRITICAL CONTEXT This task was previously marked 'done' but the audit could not verify the work actually landed on main. The original work may have been: - Lost to an orphan branch / failed push - Only a spec-file edit (no code changes) - Already addressed by other agents in the meantime - Made obsolete by subsequent work **Before doing anything else:** 1. **Re-evaluate the task in light of CURRENT main state.** Read the spec and the relevant files on origin/main NOW. The original task may have been written against a state of the code that no longer exists. 2. **Verify the task still advances SciDEX's aims.** If the system has evolved past the need for this work (different architecture, different priorities), close the task with reason "obsolete: " instead of doing it. 3. **Check if it's already done.** Run `git log --grep=''` and read the related commits. If real work landed, complete the task with `--no-sha-check --summary 'Already done in '`. 4. **Make sure your changes don't regress recent functionality.** Many agents have been working on this codebase. Before committing, run `git log --since='24 hours ago' -- ` to see what changed in your area, and verify you don't undo any of it. 5. **Stay scoped.** Only do what this specific task asks for. Do not refactor, do not "fix" unrelated issues, do not add features that weren't requested. Scope creep at this point is regression risk. If you cannot do this task safely (because it would regress, conflict with current direction, or the requirements no longer apply), escalate via `orchestra escalate` with a clear explanation instead of committing.

Completion Notes

Auto-completed by supervisor after successful deploy to main

Git Commits (16)

[Atlas] Add daily dedup scanner systemd timer [task:t-auto-dedup-cron]2026-04-20

[Docs] Add t_auto_dedup_cron spec: scheduled dedup scanning [task:t-auto-dedup-cron]2026-04-20

[Atlas] Add daily dedup scanner systemd timer [task:t-auto-dedup-cron]2026-04-20

[Docs] Add t_auto_dedup_cron spec: scheduled dedup scanning [task:t-auto-dedup-cron]2026-04-20

[Atlas] Add daily dedup scanner systemd timer [task:t-auto-dedup-cron]2026-04-20

[Docs] Add t_auto_dedup_cron spec: scheduled dedup scanning [task:t-auto-dedup-cron]2026-04-20

[Atlas] Add daily dedup scanner systemd timer [task:t-auto-dedup-cron]2026-04-20

[Docs] Add t_auto_dedup_cron spec: scheduled dedup scanning [task:t-auto-dedup-cron]2026-04-20

[Atlas] Add daily dedup scanner systemd timer [task:t-auto-dedup-cron]2026-04-20

[Docs] Add t_auto_dedup_cron spec: scheduled dedup scanning [task:t-auto-dedup-cron]2026-04-20

[Atlas] Add daily dedup scanner systemd timer [task:t-auto-dedup-cron]2026-04-20

[Docs] Add t_auto_dedup_cron spec: scheduled dedup scanning [task:t-auto-dedup-cron]2026-04-20

[Atlas] Add daily dedup scanner systemd timer [task:t-auto-dedup-cron]2026-04-20

[Docs] Add t_auto_dedup_cron spec: scheduled dedup scanning [task:t-auto-dedup-cron]2026-04-20

[Atlas] Add daily dedup scanner systemd timer [task:t-auto-dedup-cron]2026-04-20

[Docs] Add t_auto_dedup_cron spec: scheduled dedup scanning [task:t-auto-dedup-cron]2026-04-20

Spec File

Goal

Add a daily cron job running artifact_dedup_agent.run_full_scan() on a recurring schedule to continuously detect and flag artifact sprawl as the knowledge graph and wiki grow.

Acceptance Criteria

☑ scripts/recurring_dedup_pipeline.py exists and calls run_full_scan()

☑ Pipeline is idempotent (skips existing pending recommendations)

☑ Cron entry is documented in the script docstring

☑ Spec file created

Approach

The recurring dedup pipeline was implemented in prior task work (commit 85a9f67ec, now on origin/main as part of task 6493344d_4ce). This task creates the spec file documenting the architecture.

The pipeline:

Calls run_full_scan() which scans hypotheses, wiki pages, gaps, and artifacts

Uses high thresholds (hypothesis=0.42, wiki=0.60, gaps=0.55) to minimize false positives

Writes recommendations to dedup_recommendations table as status=pending

Auto-classifies by confidence tier: auto-approve (>=0.95), human_review (0.8-0.95), auto-reject (<0.8)

Executes approved merges in batch

Cron setup (documented in script docstring):

crontab -l 2>/dev/null; echo "0 */6 * * * cd /home/ubuntu/scidex && python3 scripts/recurring_dedup_pipeline.py >> /var/log/scidex/dedup_pipeline.log 2>&1" | crontab -

Dependencies

t-auto-dedup-cron (this task): spec file only
6493344d_4ce (completed): actual pipeline implementation

2026-04-20 22:15 UTC — Slot minimax:61

Audited: prior work (85a9f67ec) already on origin/main via task 6493344d_4ce
scripts/recurring_dedup_pipeline.py confirmed present and functional
Supplemental work: added scidex-dedup-scanner.{service,timer} as systemd-native daily trigger

- Service runs recurring_dedup_pipeline.py (same pipeline as cron, wrapped in systemd supervision)
- Timer fires daily at midnight local with OnBootSec=10min and Persistent=true
- Follows same pattern as existing scidex-gap-scanner.timer and scidex-pubmed-pipeline.timer

Result: Done — systemd timer adds supervised daily deduplication scanning alongside existing cron entry

Work Log

2026-04-20 21:30 UTC — Slot minimax:63

Audited task: prior work (85a9f67ec) already on origin/main
scripts/recurring_dedup_pipeline.py exists on origin/main
Pipeline calls run_full_scan(), auto_review_pending(), execute_approved_merges()
Missing only: spec file (audit reopened)
Creating spec file now
Result: Done — spec file created

Sibling Tasks in Quest (Artifact Lifecycle) ↗

✓Lifecycle schema migrationP80claude

✓Lifecycle state machineP80

✓Git-like artifact versioningP75

✓Deduplication agentP75

✓Forward reference resolutionP70

✓Artifact governance dashboardP70

✓Event bus lifecycle eventsP65

✓Dependency-aware version propagationP55