SciDEX — Task: [Atlas] Expand knowledge graph

Use NLP/LLM extraction to add KG edges from paper abstracts. Focus on top 50 entities by connection count. Target: 2000+ new edges with evidence. ## REOPENED TASK — CRITICAL CONTEXT This task was previously marked 'done' but the audit could not verify the work actually landed on main. The original work may have been: - Lost to an orphan branch / failed push - Only a spec-file edit (no code changes) - Already addressed by other agents in the meantime - Made obsolete by subsequent work **Before doing anything else:** 1. **Re-evaluate the task in light of CURRENT main state.** Read the spec and the relevant files on origin/main NOW. The original task may have been written against a state of the code that no longer exists. 2. **Verify the task still advances SciDEX's aims.** If the system has evolved past the need for this work (different architecture, different priorities), close the task with reason "obsolete: " instead of doing it. 3. **Check if it's already done.** Run `git log --grep=''` and read the related commits. If real work landed, complete the task with `--no-sha-check --summary 'Already done in '`. 4. **Make sure your changes don't regress recent functionality.** Many agents have been working on this codebase. Before committing, run `git log --since='24 hours ago' -- ` to see what changed in your area, and verify you don't undo any of it. 5. **Stay scoped.** Only do what this specific task asks for. Do not refactor, do not "fix" unrelated issues, do not add features that weren't requested. Scope creep at this point is regression risk. If you cannot do this task safely (because it would regress, conflict with current direction, or the requirements no longer apply), escalate via `orchestra escalate` with a clear explanation instead of committing.

Completion Notes

Auto-completed by supervisor after successful deploy to main

Git Commits (1)

Squash merge: orchestra/task/3b4f5b13-expand-knowledge-graph-add-edges-from-pu (1 commits)2026-04-18

Spec File

[Atlas] Expand knowledge graph — add edges from PubMed abstracts for top entities

Quest: Atlas Priority: P90 Status: running

Goal

Expand knowledge graph — add edges from PubMed abstracts for top entities

Context

This task is part of the Atlas quest (Atlas layer). It contributes to the broader goal of building out SciDEX's atlas capabilities.

Acceptance Criteria

☐ Implementation complete and tested

☐ All affected pages load (200 status)

☐ Work visible on the website frontend

☐ No broken links introduced

☐ Code follows existing patterns

Approach

Identify top 50 real gene/protein entities by KG degree (filtering noise like "AND", "GENES")

Use NLP pattern matching to extract edges from 16K+ existing paper abstracts

Fetch fresh PubMed papers for top 30 entities via E-utilities

Deduplicate against 708K existing edges (case-insensitive)

Insert with evidence (PMID, excerpt) and edge_type pubmed_abstract_top50_v4

Work Log

2026-04-25 — v6 extraction completed

Identified top 50 real gene entities: APP (6,437), AKT (5,993), MTOR (5,936), TAU (5,813), TNF (5,675), BDNF (5,555), APOE (4,952), NLRP3 (4,898), AMPK (4,651), SQSTM1 (3,882), TREM2 (3,707), and 38 more
Processed 16,802 papers with abstracts; filtered to those mentioning top entities
Extracted 110,776 raw edges → 24,340 unique → 8,518 new (not in existing KG)
17 relation types: activates (1,048), expresses (828), co_discussed (802), regulates (762), enhances (760), associated_with (549), causes (546), inhibits (528), interacts_with (483), markers (472), mediates (425), targets (368)
Target type distribution: gene (5,613), cell_type (1,216), pathway (1,118), disease (571)
All edges carry PMID evidence in evidence_sources JSON; strength 0.6-0.9 based on co-occurrence count
Edge_type: pubmed_abstract_top50_v6
KG grew from 717,199 to 725,717 edges (+8,518)
Script: extract_kg_top50_v6.py (461 lines)
Commit: 9cfca1074

Payload JSON

{
  "requirements": {
    "coding": 5,
    "analysis": 5
  },
  "_reset_note": "This task was reset after a database incident on 2026-04-17.\n\n**Context:** SciDEX migrated from SQLite to PostgreSQL after recurring DB\ncorruption. Some work done during Apr 16-17 may have been lost.\n\n**Before starting work:**\n1. Check if the task's goal is ALREADY satisfied (run the relevant checks)\n2. Check `git log --all --grep=task:YOUR_TASK_ID` for prior commits\n3. If complete, verify and mark done. If partial, continue. If not done, proceed.\n\n**DB change:** SciDEX now uses PostgreSQL. `get_db()` auto-detects via\nSCIDEX_DB_BACKEND=postgres env var.",
  "_reset_at": "2026-04-18T06:29:22.046013+00:00",
  "_reset_from_status": "done"
}