[Atlas] Scheduled dedup scanning

Goal

Add a daily cron job running artifact_dedup_agent.run_full_scan() on a recurring schedule to continuously detect and flag artifact sprawl as the knowledge graph and wiki grow.

Acceptance Criteria

Approach

The recurring dedup pipeline was implemented in prior task work (commit 85a9f67ec, now on origin/main as part of task 6493344d_4ce). This task creates the spec file documenting the architecture.

The pipeline:

Calls run_full_scan() which scans hypotheses, wiki pages, gaps, and artifacts

Uses high thresholds (hypothesis=0.42, wiki=0.60, gaps=0.55) to minimize false positives

Writes recommendations to dedup_recommendations table as status=pending

Auto-classifies by confidence tier: auto-approve (>=0.95), human_review (0.8-0.95), auto-reject (<0.8)

Executes approved merges in batch

Cron setup (documented in script docstring):

crontab -l 2>/dev/null; echo "0 */6 * * * cd /home/ubuntu/scidex && python3 scripts/recurring_dedup_pipeline.py >> /var/log/scidex/dedup_pipeline.log 2>&1" | crontab -

Dependencies

• t-auto-dedup-cron (this task): spec file only
• 6493344d_4ce (completed): actual pipeline implementation

2026-04-20 22:15 UTC — Slot minimax:61

• Audited: prior work (85a9f67ec) already on origin/main via task 6493344d_4ce
• scripts/recurring_dedup_pipeline.py confirmed present and functional
• Supplemental work: added scidex-dedup-scanner.{service,timer} as systemd-native daily trigger

- Service runs recurring_dedup_pipeline.py (same pipeline as cron, wrapped in systemd supervision)
- Timer fires daily at midnight local with OnBootSec=10min and Persistent=true
- Follows same pattern as existing scidex-gap-scanner.timer and scidex-pubmed-pipeline.timer

• Result: Done — systemd timer adds supervised daily deduplication scanning alongside existing cron entry

Work Log

2026-04-20 21:30 UTC — Slot minimax:63

• Audited task: prior work (85a9f67ec) already on origin/main
• scripts/recurring_dedup_pipeline.py exists on origin/main
• Pipeline calls run_full_scan(), auto_review_pending(), execute_approved_merges()
• Missing only: spec file (audit reopened)
• Creating spec file now
• Result: Done — spec file created

Tasks using this spec (1)

Scheduled dedup scanning

File: t_auto_dedup_cron_spec.md

Modified: 2026-04-28 03:24

Size: 2.5 KB