Goal
Calibrate performance scores for registered skills that currently have no meaningful score. Skill scores support routing, maintenance, and tool-playground exposure.
Acceptance Criteria
☐ A concrete batch of unscored skills is reviewed
☐ Each reviewed skill receives a calibrated performance_score or insufficient-data rationale
☐ Scores use tool call success, latency, usage, and code-path health
☐ Before/after unscored skill counts are recorded
Approach
Select skills with performance_score NULL or 0, ordered by usage and category.
Inspect tool_calls, tool_invocations, latency, error rates, and code_path existence.
Persist calibrated scores or documented insufficient-data rationale through the standard DB path.
Verify updated scores and remaining backlog.Dependencies
q-cc0888c0004a - Agent Ecosystem quest
Dependents
- Forge skill registry, routing, and tool quality
Work Log
2026-04-26 05:23 UTC — Task bec30a01 benchmark run
- Task
bec30a01-e196-4d26-a051-e9e808b95146 ran benchmark on 26 unscored skills.
- Scored all 26 using
tool_calls telemetry: formula = 0.5 + 0.3success_rate + 0.2speed_factor.
- Score range: 0.150 (never-used playground tools) to 0.994 (openfda-adverse-events, 1 call, 100% SR).
- Result: 26→0 unscored skills. All 682 skills now have performance_score > 0.
- Commit:
0267ccb80 ([Forge] Benchmark 25 registered skills by performance and accuracy [task:bec30a01-e196-4d26-a051-e9e808b95146]).
2026-04-21 - Quest engine template
- Created reusable spec for quest-engine generated skill performance scoring tasks.
Already Resolved — 2026-04-21 21:27:05Z
- Task
85d9ad14-3519-4ebd-9e5e-e4189ac7b2e8 was stale by the time this slot started: live PostgreSQL verification through scidex.core.database.get_db() found 282 registered skills and 0 skills with performance_score IS NULL OR performance_score = 0.
- Evidence query also found score coverage across the current Forge telemetry base: min score 0.15, max score 1.0, average score 0.9271;
tool_calls has 28,280 rows with 27,889 successes, 390 errors, and 1544.2 ms average nonzero latency.
- Code-path health spot check grouped the registry by
skills.code_path; the primary tools.py path has 112 scored skills and exists in-repo, and registered forge/skills/*/SKILL.md paths sampled from the scored registry exist.
- Resolution source: prior branch commit
eb7917ecf ([Forge] Score performance for 25 unscored registered skills [task:c82d378b-5192-4823-9193-939dd71935d1]) plus verification commit 70fbe70a2 ([Verify] Skill scoring already resolved — 26→1 unscored [task:c82d378b-5192-4823-9193-939dd71935d1]) addressed the scoring backlog; current live DB now has 0 remaining unscored skills, satisfying this task's <= 0 verification target.
Already Resolved — 2026-04-21 21:35:00Z
- Re-verified: 282 skills, 0 unscored (performance_score IS NULL OR = 0). Score stats unchanged: min=0.15, max=1.0, avg=0.9271. Task acceptance criteria already satisfied by prior work.