Assign quality scores to unscored world-model artifacts so Atlas curation and Exchange artifact-quality markets have usable signals. Scores should reflect scientific utility, evidence strength, reproducibility, novelty, and reuse value.
quality_score values between 0 and 1quality_score IS NULL OR quality_score = 0.415b277f-03b - Atlas questquality_status — acceptance criterion: 50 receive values or insufficient-data rationale.quality_score IS NOT NULL for all 14,027, but quality_status column was never populated for these rows (separate concern from quality_score).pass (qs 0.64-0.96), ok (qs 0.09-1.0), flagged (qs 0.10-0.57), fail (qs 0.20-0.36), scored (qs 0.47-0.72).scripts/score_quality_status.py — selects top 50 by quality_score DESC, assigns status by calibrated thresholds (qs>=0.8->pass, qs>=0.4->ok, qs<0.4->flagged), persists rationale in metadata.quality_scoring.quality_status='pass' (qs range 0.900-0.990)./atlas.html and /api/artifacts return 200.7f2bef796 pushed to orchestra/task/3854fa92-verify-quality-status-for-50-artifacts-m.Result: PASS
Verified by: MiniMax-M2 via task 3854fa92-f632-4cba-be41-396420da7151
7f2bef796 — [Atlas] Assign quality_status to 50 high-quality artifacts missing review state [task:3854fa92-f632-4cba-be41-396420da7151]quality_score was already populated for all 14,027 rows; the gap was that quality_status was never set.open_question (7,838), paper_figure (3,531), figure (1,896) that mostly have qs=0.500 — no compelling signal to differentiate them algorithmically without per-artifact review.qs>0.7 for pass, qs 0.4-0.7 for ok, etc., but full coverage of remaining types needs human-in-the-loop triage.807d42c0-ad71-4fbb-b47b-e3cfcb7d2edc.artifacts already had 0 unscored rows before work (quality_score IS NULL OR quality_score = 0), including 0 unscored paper rows; all 520 paper artifacts had scores in [0.09, 1.0].56a7341b30ccbffc5594b3edfe6ad22ca07208b9 ([Atlas] Score 30 paper artifacts with quality scoring [task:ebade91a-4f4c-4088-b7f0-9389b70ad12e]) that added scripts/score_paper_artifacts.py, but current paper artifact metadata had 0 rows with quality rationale text.scripts/score_paper_artifacts.py to persist metadata.quality_scoring with task ID, timestamp, scoring formula, dimension scores, input signals, dimension rationales, and a plain-language rationale. Replaced JSONB ? predicates with jsonb_exists(...) because the SciDEX cursor rewrites ? as a placeholder.python3 scripts/score_paper_artifacts.py: scored/rationalized 30 paper artifacts; unscored artifact count was already 0 before and remained 0 after; task-tagged rationale rows = 30.paper artifacts with metadata.quality_scoring.task_id = 807d42c0-ad71-4fbb-b47b-e3cfcb7d2edc and quality_score BETWEEN 0 AND 1; score range 0.475-0.683. Sample rationale rows included paper-30745308, paper-33012345, paper-33234567, paper-33504552, and paper-23283301.bcf54dd86 ([Atlas] Persist paper artifact quality rationales [task:807d42c0-ad71-4fbb-b47b-e3cfcb7d2edc]) is the landing commit on this branch.quality_score IS NULL AND artifact_type IN ('paper','paper_figure')./home/ubuntu/Orchestra/data/claude_creds/max_outlook/session-env/). All Bash commands fail before execution; database queries and git operations are blocked.score_paper_artifacts.py (PostgreSQL, paper only), score_fig_artifacts.py (SQLite, retired), score_artifacts.py (SQLite, retired).scripts/score_paper_and_figure_artifacts.py — a new PostgreSQL-based script covering both paper and paper_figure artifact types with the task-specified dimensions: content_completeness, scientific_relevance, citation_signal, data_richness (mean = quality_score). Persists per-artifact rationale in metadata.quality_scoring.python3 scripts/score_paper_and_figure_artifacts.py from the repo root, then commit with [Atlas] Score 50 paper/figure artifacts [task:401087fb-e089-4906-9114-4089d05db250].Context: Task e98f5d97-9571-43ec-928c-425270079951 — "[Senate] Review 50 artifacts with empty quality_status for lifecycle readiness"
Findings:
quality_status IS NULL with quality_score > 0 (no unscored, just un-reviewed)quality_score values are all default-banded: open_question/paper_figure/evidence/kg_edge all = 0.500; figure all = 0.700; notebooks = 0.35-0.82 range; hypotheses = 0.46-0.89 rangequality_score > 0 — the gap is the explicit quality_status review state, not the score itselfscripts/assign_quality_status.py — derives quality_status from quality_score, artifact_type, and lifecycle_state using bands consistent with existing scored artifactsquality_status = 'ok' with qs=0.7quality_status = 'ok' with metadata.quality_status_assignment.task_id = e98f5d97-9571-43ec-928c-425270079951scripts/assign_quality_status.py in larger batches (500-1000 per run) to avoid long transactions.quality_score = 0.500 — this is a generic placeholder that doesn't reflect actual evidence strength. A Senate task should either (a) backfill meaningful scores for open questions linked to actual hypotheses/papers, or (b) mark them as a distinct quality_status = 'incomplete' category pending evidence linkage.quality_status = 'ok' with quality_score < 0.4 — this validates the scoring bands are internally consistent with the existing status schema.Evidence run: python3 -c "from scidex.core.database import get_db; db = get_db(); cur = db.cursor(); cur.execute('SELECT COUNT(*) FROM artifacts WHERE quality_score IS NULL AND artifact_type IN (%s, %s)', ('paper', 'paper_figure')); print('Unscored:', cur.fetchone()[0])" → returned 0
Acceptance criteria verified:
807d42c0 persisted rationale via metadata.quality_scoringSummary: All paper and paper_figure artifacts already have quality_score values; no unscored rows exist to score. Task is satisfied by prior scoring work on main.
scripts/set_quality_status_batch.py — derives quality_status from quality_score + lifecycle_state using bands consistent with existing scored artifact distribution:metadata.quality_status_rationale recording task_id, timestamp, rule_reason, and signals (quality_score, lifecycle_state, origin_type, provenance_chain_length).3fdae0cc2 pushed to branch.Task: Rank 25 analyses missing world-model impact scores (acceptance: ≤ 156 remaining).
Before state: 159 analyses with world_model_rank IS NULL OR world_model_rank = 0 (26 NULL, 133 zero).
Prior work (not merged): Commit c7241203a on direct-origin/orchestra/task/ce3182a7-* had ranked 25 completed analyses, reducing 181→156. The merge was blocked by rate_limit_retries_exhausted:glm. DB changes from that run persisted (181→159 at start of this cycle, delta of 3 from new analyses created between runs).
Script: scripts/rank_analyses_world_model.py — 5-factor weighted scoring formula:
0.30 × kg_impact (kg_impact_score / 500)0.25 × hyp_quality (avg composite_score on linked hypotheses, or avg_debate_quality×0.8 fallback)0.20 × debate_depth (avg_debate_quality × min(debate_count/3, 1))0.15 × gap_addressed (1.0 if gap_id set, else 0.0)0.10 × research_depth (min(total_hyps_generated/10, 1.0))analyses.metadata['world_model_ranking'].Actions this cycle:
completed non-corrupted analyses: ranks 0.416–0.510; 159→139.archived analyses (top by hypothesis/debate count): legacy-pre-pipeline-import-v1 (116 hyps, rank 0.295), test-hypothesis-fixtures-v1 (23 hyps, rank 0.337), plus 3 gap analyses; 139→134.Remaining 134 breakdown (documented insufficient-data rationale):
archived: historical analyses — prior impact captured; lower priority for future cyclesabandoned: analysis was abandoned before completion; insufficient signalsfailed: execution failure; no reliable hypothesis/debate dataactive + 1 running: in-progress; ranking prematureworld_model_rank (≤ 156 threshold ✓).Acceptance criteria:
world_model_rank (20 completed + 5 archived){
"requirements": {
"analysis": 7,
"reasoning": 7
}
}