[Artifacts] Dashboard artifact type: living web views that auto-update from data sources done coding:7 reasoning:6

[Artifacts] Dashboard artifact type

Goal

Dashboards are living artifacts — web pages that stay current as their underlying data changes. Unlike static figures or reports, dashboards bind to data sources (SQL queries, artifact references, API endpoints) and re-render automatically. They can be displayed standalone or embedded in authored papers, creating truly dynamic scientific documents.

Design

Dashboard vs Other Artifacts

Feature	Figure	Notebook	Authored Paper	Dashboard
Static/Live	Static	Static (re-runnable)	Semi-live (embeds)	Live (auto-refresh)
Data binding	None	Code cells	Artifact embeds	Data source queries
Rendering	Image	HTML (nbconvert)	HTML + embeds	HTML + live data
Update trigger	Manual	Re-execute	Edit content	Auto (interval/event)

Metadata Schema

{
  "template_html": "<div class='dashboard'>{{data:paper_count}}...</div>",
  "data_sources": [
    {
      "key": "paper_count",
      "type": "sql_query",
      "query": "SELECT COUNT(*) as value FROM papers",
      "description": "Total papers in SciDEX"
    },
    {
      "key": "hypothesis_trend",
      "type": "sql_query", 
      "query": "SELECT date(created_at) as date, COUNT(*) as count FROM hypotheses GROUP BY date(created_at) ORDER BY date DESC LIMIT 30",
      "description": "Hypothesis creation trend (30 days)"
    },
    {
      "key": "top_hypothesis",
      "type": "artifact_ref",
      "artifact_id": "hypothesis-h-seaad-v4-26ba859b",
      "description": "Current top-scoring hypothesis"
    },
    {
      "key": "kg_stats",
      "type": "api_endpoint",
      "url": "/api/status",
      "jq_filter": ".knowledge_graph",
      "description": "KG statistics from status API"
    }
  ],
  "refresh_interval_seconds": 300,
  "last_rendered_at": "2026-04-03T12:00:00",
  "rendered_html": "<div class='dashboard'>15025...</div>",
  "document_type": "dashboard"
}

Data Source Types

Type	Description	Execution
sql_query	SQLite query against PostgreSQL	Execute query, return rows as JSON
artifact_ref	Reference to another artifact	Fetch latest version, extract display fields
api_endpoint	Internal API endpoint	HTTP GET, optional jq-style filtering
kg_query	Knowledge graph query	Cypher-like query returning nodes/edges

REST Endpoints

• GET /api/dashboards — list all dashboard artifacts
• GET /api/dashboard/{id} — dashboard detail with current rendered HTML
• GET /api/dashboard/{id}/refresh — trigger re-render and return fresh HTML
• GET /dashboard/{id} — standalone dashboard page (auto-refreshing)

Rendering Pipeline

Load dashboard artifact metadata

For each data source, evaluate query/fetch and get result

Replace {{data:KEY}} placeholders in template_html with formatted results

Store rendered_html in metadata for caching

Update last_rendered_at timestamp

Return rendered HTML

Auto-Refresh

• Dashboards with refresh_interval_seconds > 0 are periodically re-rendered
• Frontend uses JavaScript setInterval to poll /api/dashboard/{id}/refresh
• Server-side: background task or on-demand rendering with cache TTL

Acceptance Criteria

Dependencies

• a17-19-TYPE0001 (dashboard type registration)
• a17-30-DREF0001 (data source refresh engine)

Dependents

• d16-26-DDSH0001 (demo: knowledge growth dashboard)

Work Log

• 2026-04-04 UTC: Implementation complete.

- Updated register_dashboard() in artifact_registry.py to use spec interface (template_html, data_sources, refresh_interval_seconds). Legacy query_config format auto-converted for backward compatibility.
- Added render_dashboard() function: evaluates sql_query/artifact_ref/api_endpoint sources, replaces {{data:KEY}} placeholders, caches rendered_html and last_rendered_at.
- Added 4 API endpoints to api.py: GET /api/dashboards, GET /api/dashboard/{id}, GET /api/dashboard/{id}/refresh, GET /dashboard/{id} (standalone with auto-refresh JS).
- Updated dashboard embed rendering to use pre-rendered HTML when available.
- Fixed pre-existing syntax errors in api.py (nested f-string with backslash at line 3635, broken f-string format at line 3656).

• 2026-04-26 UTC: Critical bug fix — psycopg3 jsonb auto-parsing.

- Root cause: psycopg3 returns jsonb columns as Python dicts, but render_dashboard() and other functions called json.loads() on already-parsed dicts, caught the TypeError silently, and overwrote metadata with empty dict — destroying template_html and data_sources on every render.
- Fixed metadata parsing in: artifact_registry.py render_dashboard() (line ~1060), artifact_registry.py dataset query (line ~1515), dashboard_engine.py render_dashboard() (line ~221), dashboard_engine.py list_dashboards() (line ~417), dashboard_engine.py create_dashboard_page_html() (line ~456).
- Fixed create_artifact_version() metadata serialization (line ~2802): psycopg can't adapt a Python dict for jsonb — now always serializes to JSON string.
- Updated API /api/dashboard/{id}/refresh to use dashboard_engine.render_dashboard() for richer rendering (TTL caching, format specifiers, error reporting).
- Restored metadata for 4 corrupted dashboards (knowledge-growth-monitor, test-001, 6f6dd971, quest-meta) whose template_html and data_sources were wiped by the bug.

• 2026-04-26 UTC: Second pass — legacy query_config fallback in dashboard_engine.

- dashboard_engine.render_dashboard() was missing the query_config → data_sources conversion that artifact_registry.py has. Dashboards whose metadata was wiped during the psycopg3 bug have neither data_sources nor template_html fields in current DB state — they cannot be rendered by dashboard_engine even with the conversion in artifact_registry.py (the engine uses a different code path).
- Added legacy query_config panel → data_sources converter to dashboard_engine.render_dashboard() (lines 254–272) so dashboards that originally used the query_config/panels format can still render.
- Committed as f8b05a4d9 to branch orchestra/task/a17-28-D-dashboard-artifact-type-living-web-views.
- Updated /dashboard/{id} standalone page to use dashboard_engine for initial rendering.
- Restored metadata for 4 corrupted dashboards (knowledge-growth-monitor, test-001, 6f6dd971, quest-meta) whose template_html and data_sources were wiped by the bug.
- Verified end-to-end: rendering produces correct HTML with live data, versioning creates new versions with proper parent linkage.