# medbench.kr: guide for AI agents

Version: p0 (2026-10-29). Base URL: `https://medbench.kr`

medbench.kr evaluates medical AI models on open datasets from Seoul National University Hospital (SNUH), once per model and task, and publishes scores, per-item predictions and exact versions so that nobody has to repeat the same experiment. Everything on the website is also available as JSON under `/v1`. Machine-readable spec: `https://medbench.kr/v1/openapi.json`.

## Rule one: look up before you run

Before you evaluate a model on a medbench task yourself, ask whether the result already exists:

```bash
curl "https://medbench.kr/v1/lookup?model=google/medgemma-27b&task=clinicalqa@0"
# {"status": "evaluated", "result": {...}}   or   {"status": "not_evaluated", "result": null}
```

`model` is the model id shown in results, `<developer>/<model>` (`google/medgemma-27b`, `anthropic/claude-haiku-5.5`). `task` accepts `slug@version` or `slug` (latest version). If the result exists, cite it and download its predictions instead of spending compute.

## Endpoints

All responses are JSON. No token is needed in p0.

| Route | Returns |
|---|---|
| `GET /v1/datasets` | Datasets with license, license URL and citation |
| `GET /v1/tasks` | Tasks: slug, version, dataset, kind, primary metric, preview flag, number of results |
| `GET /v1/tasks/{slug}` | Full task specification (`?version=` for a specific version) |
| `GET /v1/leaderboard?task=<slug[@version]>&model_type=&limit=` | Ranked rows for one task |
| `GET /v1/results?dataset=&task=&model=&model_type=` | Results matching all given filters |
| `GET /v1/results/{run_id}` | One result: all metrics with 95% CIs, configuration, versions, cost, BibTeX, predictions URL |
| `GET /v1/lookup?model=&task=` | Whether a model has been evaluated on a task |
| `GET /v1/stats` | Counts, total cost, number of reuses |
| `GET /v1/papers?status=validated\|queued` | Papers the paper agent validated or has queued, with task, metric and rank |
| `GET /v1/papers/{doi}` | One paper: our summary, model at a glance, results on SNUH data (DOIs contain slashes; use them as they are) |
| `GET /v1/agent/now` | What the paper agent is doing right now: paper, stage (1-7), recent log lines, queue |
| `GET /artifacts/{run_id}/predictions.parquet` | Per-item predictions for one result |

Examples:

```bash
# Leaderboard for one task, LLMs only
curl "https://medbench.kr/v1/leaderboard?task=clinicalqa@0&model_type=llm"

# One result with metrics, versions and BibTeX
curl "https://medbench.kr/v1/results/<run_id>"

# Per-item predictions (columns: instance_id, prediction or score, label, correct)
curl -o preds.parquet "https://medbench.kr/artifacts/<run_id>/predictions.parquet"
```

A leaderboard row looks like:

```json
{"rank": 1, "run_id": "0123456789ab", "task": "clinicalqa@0",
 "model": {"id": "google/medgemma-27b", "name": "MedGemma 27B", "type": "llm", "developer": "Google"},
 "tier": "standard", "metric": {"name": "accuracy", "value": 0.71, "ci_low": 0.68, "ci_high": 0.74},
 "n": 1045, "parse_rate": 0.99, "cost_usd": 0.21, "eval_date": "2026-10-29",
 "url": "https://medbench.kr/r/0123456789ab"}
```

## Tasks in p0

| Task | Data | What is scored |
|---|---|---|
| `clinicalqa@0` | ClinicalQA, 1,045 Korean KMLE-level five-option questions | Accuracy; unparsable answers count as wrong |
| `vitaldb-ioh@0` | VitalDB, numeric mean arterial pressure every 2 s | AUROC for hypotension (MAP < 65 mmHg for ≥ 60 s) in the 5 minutes after a 5-minute window; patient-level split |
| `inspire-mortality@0` | INSPIRE 1.3, one operation per patient: age, sex, emergency, BMI, anesthesia duration, ASA class, 13 preoperative labs | AUROC for in-hospital death within 30 days after leaving the operating room; cohort, inputs and train/test split from the INSPIRE authors' `gbm_mortality.py` (deviations in the spec) |
| `vitaldb-arrhythmia-abnormal@0` | VitalDB Arrhythmia Database, 5-second ECG lead II segments at 500 Hz | AUROC for abnormal rhythm (any rhythm label other than normal sinus rhythm), as in the dataset paper's technical validation; Noise not scored; patient-level split shared with other VitalDB tasks |

All are **preview** versions and may change. A result always names the exact task version and specification hash it was computed on.

## Datasets, licenses and citation

- **ClinicalQA**: CC BY 4.0 on KHDP (upstream Apache-2.0). Cite: SNUH-HARI. ClinicalQA: Korean Medical Knowledge QA Benchmark. Seoul National University Hospital Healthcare AI Research Institute, 2025. doi:10.23421/2026.0800. It has been public on Hugging Face since 2025-04-04; models trained after that may have seen it.
- **VitalDB**: CC BY 4.0. Cite: Lee HC, Park Y, Yoon SB, Yang SM, Park D, Jung CW. VitalDB, a high-fidelity multi-parameter vital signs database in surgical patients. Sci Data. 2022;9:279. doi:10.1038/s41597-022-01411-5.
- **INSPIRE**: CC BY 4.0 on KHDP (`inspire` 1.3, doi:10.13026/4evs-wq50). Cite: Lim L, Lee H, Jung CW, Sim D, Borrat X, Pollard TJ, Celi LA, Mark RG, Vistisen ST, Lee HC. INSPIRE, a publicly available research dataset for perioperative medicine. Sci Data. 2024;11:655. doi:10.1038/s41597-024-03517-4. Measurements are binned to 5th-percentile values, so inputs are ordinal categories.
- **VitalDB Arrhythmia Database**: CC BY 4.0. Cite: Eun D, Shim K, Lee H, Lim Y, Lim H, Lee H, Lee HC. VitalDB Arrhythmia Database: An Anesthesiologist-Validated Large-Scale Intraoperative Arrhythmia Dataset with Beat and Rhythm Labels (version 1.0.0). PhysioNet. 2026. doi:10.13026/axd6-wm13. The ECG waveforms come from VitalDB (cite VitalDB as well).

medbench results and per-item predictions are released under CC BY 4.0. When you use a medbench result, cite the dataset and the result (BibTeX is in `/v1/results/{run_id}`).

## Papers and the live agent

An agent reads new papers with released code and weights (npj Digital Medicine first, arXiv next), picks the open SNUH dataset that fits, runs the authors' model and publishes the result as a preview with tier `reference`. When no existing task fits, it defines one from the paper's methods; such tasks are `version 0`, marked preview, and list any `deviations` from the paper in their spec. Papers it could not run are not listed. Watch it at `https://medbench.kr/live`.

## Tags (field `tier`)

Each result says who ran the model:

- **official**: run from what the model's authors submitted (not open yet).
- **standard**: run by medbench with the standard prompt and harness, identical for every model on the task.
- **agent**: run by the medbench paper agent with the authors' released weights, on open SNUH data. Not a verdict on the paper: the agent's setup can differ from the authors'.
- **baseline**: a simple comparison model built by medbench (majority class, logistic regression).

## Not yet available

Submitting models, API tokens and the `medbench` CLI are planned. Until then, to request an evaluation, open an issue or contact the maintainers listed on https://medbench.kr/about.

## Errors

Errors are JSON with a machine-readable code: `{"error": "task_not_found", "message": "..."}`. Codes: `task_not_found`, `run_not_found`.

medbench.kr is an independent project using open SNUH datasets. It is not an official service of Seoul National University Hospital.