[Senate] Design resource allocation economics framework

Goal

Define the Senate-layer economics framework that governs how SciDEX allocates scarce execution resources across hypotheses, entities, gaps, and analyses. The framework should make resource decisions legible, evidence-based, and compatible with the Exchange and Atlas layers so high-value work receives more compute while weak or low-yield work is throttled.

Acceptance Criteria

Approach

Audit the live PostgreSQL schema and current runtime code to see which economics tables already exist and which columns are already depended on.

Add the missing budget table and task-contract compatibility columns using an idempotent migration that preserves the broader deployed schemas.

Replace the stale placeholder spec with an in-repo design document that explains the allocation formula, governance flow, market weighting, and ROI review loop.

Run the migration and inspect PostgreSQL metadata to verify the resulting table and column set.

Dependencies

• d5cc6ada-a31 — allocation proposal workflow already established and must remain compatible.
• 370890cc-17a6-4afc-803c-f3219625c49d — resource usage tracking already established and must remain compatible.
• 7dea0448787e — daily budget checks already rely on resource_usage cost rows.

Dependents

• Future Senate UI and /api/economics/* work can use resource_budgets as the canonical per-entity envelope table.
• Exchange-driven funding and Atlas ROI analytics can read the compatibility columns added here without rewriting existing consumers.

Framework

Resource Types

SciDEX allocates four first-class resource classes:

• llm_tokens: budgeted by model tier so expensive reasoning models and cheap batch models do not collapse into one pool.
• cpu_seconds / cpu_budget: used for local analysis jobs, post-processing, and background drivers.
• storage_gb_hours: used for notebooks, figures, artifacts, and cached intermediate outputs.
• api_calls: used for PubMed, Semantic Scholar, OpenAlex, Crossref, and other external providers.

The accounting model remains additive: every concrete execution emits resource_usage rows, while policy decisions write or update resource_budgets and allocation_proposals.

Budget Model

Every allocatable object is modeled as (entity_type, entity_id), where entity_type is typically one of hypothesis, knowledge_gap, analysis, or entity. Each object receives a budget envelope in resource_budgets:

• token_budget: token credits available for LLM-bound work.
• cpu_budget: compute allowance for local or external jobs.
• storage_budget_gb and api_call_budget: optional secondary caps for artifacts and external APIs.
• priority_score: policy score used to rank entities when budgets are scarce.
• market_weight and roi_score: continuously updated modifiers derived from Exchange prices and Atlas ROI metrics.

The table is deliberately budget-envelope oriented rather than usage-log oriented. Usage remains append-only in resource_usage; budgets are the editable control surface.

Allocation Scoring

The canonical allocation score is a weighted composite used to set or revise priority_score:

allocation_score = 0.35 evidence_strength + 0.25 novelty + 0.20 strategic_alignment + 0.20 community_interest

Definitions:

• evidence_strength: quality and amount of support already present in papers, debates, replication signals, or KG links.
• novelty: whether the work opens underexplored terrain rather than re-spending resources on saturated topics.
• strategic_alignment: how directly the object advances the five-layer world-model roadmap.
• community_interest: demand signal from debates, comments, votes, bounties, or review attention.

market_weight is not folded into the base score. It is applied after scoring so Exchange signals modulate, rather than fully overwrite, Senate judgment.

Governance

Resource increases are proposed through allocation_proposals. The governance workflow is:

A proposer files a request with proposer, entity_type, entity_id, requested_amount, justification, and optional votes ledger.

The proposal must include evidence-based justification grounded in one or more of: new PMIDs, increased market confidence, higher KG yield, successful benchmark results, or gap-resolution progress.

Senate review can approve, reject, or defer based on the quality of evidence and the current budget envelope.

Approved proposals update the relevant resource_budgets row; all actual spend continues to land in resource_usage.

The justification requirement is intentionally strict: a resource increase is valid only when the proposer links the requested spend to expected world-model gain.

Market Mechanism

Exchange scores influence budgets multiplicatively. A simple policy rule for a review cycle is:

effective_budget = base_budget * (1 + market_weight)

where market_weight is derived from normalized Exchange confidence or price signals. High-confidence hypotheses therefore receive proportionally more budget, but only after clearing the Senate scoring and governance thresholds above.

This keeps market signals directional rather than sovereign: speculative enthusiasm can accelerate work, but it cannot bypass evidence standards.

Dynamic Rebalancing

The Senate reviews allocations on a cadence such as daily for spend safety and weekly for strategic rebalance. The core ROI metric is:

roi_score = kg_edges_gained / tokens_spent

This can be extended with papers added, hypotheses improved, or artifact-quality gains, but KG-edge gain per token spent is the primary anchor because it maps directly to world-model expansion. During review:

• entities with strong ROI and strong market support can gain budget,
• entities with sustained low ROI can be capped or reduced,
• idle budgets can be reclaimed and reassigned.

Schema Notes

The live deployment already had broader versions of two task tables:

• resource_usage already tracked amount, created_at, cost, model, and artifact context.
• allocation_proposals already tracked requested_tokens, proposer metadata, approval state, and evidence fields.

This task therefore adds compatibility columns instead of replacing the existing schemas:

• resource_usage.amount_used mirrors amount.
• resource_usage.timestamp mirrors created_at.
• allocation_proposals.requested_amount mirrors requested_tokens.
• allocation_proposals.proposer mirrors proposer_id or proposer_agent.
• allocation_proposals.votes stores structured Senate vote records as jsonb.

Work Log

2026-04-24 08:34 PT — Slot 52

• Audited the task against the live schema before coding.
• Confirmed resource_usage and allocation_proposals already exist in PostgreSQL, but resource_budgets does not.
• Confirmed the existing spec was stale: it claimed resource_budgets existed and pointed to an external Google Doc instead of serving as the in-repo design source.
• Chosen implementation path: add only the missing budget table plus additive compatibility columns, preserving existing runtime consumers.

2026-04-24 08:42 PT — Slot 52

• Added migrations/121_resource_allocation_economics.py.
• Defined resource_budgets with token, CPU, storage, API, priority, market-weight, and ROI fields plus ranking indexes.
• Added additive compatibility columns to resource_usage (amount_used, timestamp) and allocation_proposals (proposer, requested_amount, votes) with backfills from existing canonical columns.
• Replaced the stale framework placeholder with this in-repo design spec so the repo now contains the authoritative economics policy and schema notes.

2026-04-24 08:45 PT — Slot 52

• Ran python3 migrations/121_resource_allocation_economics.py successfully against PostgreSQL.
• Verified resource_budgets now exists and inspected its full column list through information_schema.columns.
• Verified resource_usage now exposes amount_used and timestamp.
• Verified allocation_proposals now exposes proposer, requested_amount, and votes.

2026-04-24 08:52 PT — Slot 52

• Re-checked the migration against current runtime consumers in scidex/core/resource_tracker.py and scidex/senate/resource_governance.py to confirm they still rely on the original canonical columns.
• Queried PostgreSQL to verify the additive compatibility fields are fully backfilled: resource_usage.amount_used/timestamp and allocation_proposals.proposer/requested_amount/votes all returned zero missing rows.
• Confirmed origin/main still carried the stale external-doc placeholder spec, so this task remained necessary and non-duplicative.

2026-04-24 09:02 PT — Slot 52

• Re-ran python3 migrations/121_resource_allocation_economics.py to confirm the migration remains idempotent after the first successful apply.
• Re-queried information_schema.columns for resource_budgets and re-checked null counts on the new compatibility columns; all expected columns remain present and the new fields still report zero missing rows.

2026-04-24 08:29 PDT — Slot 52

• Revalidated the task against the current worktree and origin/main before commit preparation.
• Confirmed origin/main:docs/planning/specs/economics_framework_spec.md still contains the stale external-doc placeholder, while this worktree contains the in-repo framework and migration.
• Re-ran python3 migrations/121_resource_allocation_economics.py and an information_schema.columns audit; resource_budgets exists, the compatibility columns remain present, and null-count checks for amount_used, timestamp, proposer, and requested_amount all remained at zero.

File: economics_framework_spec.md

Modified: 2026-04-24 08:38

Size: 10.3 KB