# medbench.kr: guide for AI agents

Version: p0 (2026-10-29). Base URL: `https://medbench.kr`

medbench.kr evaluates medical AI models on open datasets from Seoul National University Hospital (SNUH), once per model and task, and publishes scores, per-item predictions and exact versions so that nobody has to repeat the same experiment. Everything on the website is also available as JSON under `/v1`. Machine-readable spec: `https://medbench.kr/v1/openapi.json`.

## Rule one: look up before you run

Before you evaluate a model on a medbench task yourself, ask whether the result already exists:

```bash
curl "https://medbench.kr/v1/lookup?model=medgemma-27b&task=clinicalqa@0"
# {"status": "evaluated", "result": {...}}   or   {"status": "not_evaluated", "result": null}
```

`model` accepts a medbench model slug (`kgpu/medgemma-27b`) or the provider's model id (`medgemma-27b`). `task` accepts `slug@version` or `slug` (latest version). If the result exists, cite it and download its predictions instead of spending compute.

## Endpoints

All responses are JSON. No token is needed in p0.

| Route | Returns |
|---|---|
| `GET /v1/datasets` | Datasets with license, license URL and citation |
| `GET /v1/tasks` | Tasks: slug, version, dataset, kind, primary metric, preview flag, number of results |
| `GET /v1/tasks/{slug}` | Full task specification (`?version=` for a specific version) |
| `GET /v1/leaderboard?task=<slug[@version]>&model_type=&limit=` | Ranked rows for one task |
| `GET /v1/results?dataset=&task=&model=&model_type=` | Results matching all given filters |
| `GET /v1/results/{run_id}` | One result: all metrics with 95% CIs, configuration, versions, cost, BibTeX, predictions URL |
| `GET /v1/lookup?model=&task=` | Whether a model has been evaluated on a task |
| `GET /v1/stats` | Counts, total cost, number of reuses |
| `GET /artifacts/{run_id}/predictions.parquet` | Per-item predictions for one result |

Examples:

```bash
# Leaderboard for one task, LLMs only
curl "https://medbench.kr/v1/leaderboard?task=clinicalqa@0&model_type=llm"

# One result with metrics, versions and BibTeX
curl "https://medbench.kr/v1/results/<run_id>"

# Per-item predictions (columns: instance_id, prediction or score, label, correct)
curl -o preds.parquet "https://medbench.kr/artifacts/<run_id>/predictions.parquet"
```

A leaderboard row looks like:

```json
{"rank": 1, "run_id": "0123456789ab", "task": "clinicalqa@0",
 "model": {"slug": "kgpu/medgemma-27b", "name": "MedGemma 27B", "type": "llm", "provider": "kgpu"},
 "tier": "baseline", "metric": {"name": "accuracy", "value": 0.71, "ci_low": 0.68, "ci_high": 0.74},
 "n": 1045, "parse_rate": 0.99, "cost_usd": 0.21, "eval_date": "2026-10-29",
 "url": "https://medbench.kr/r/0123456789ab"}
```

## Tasks in p0

| Task | Data | What is scored |
|---|---|---|
| `clinicalqa@0` | ClinicalQA, 1,045 Korean KMLE-level five-option questions | Accuracy; unparsable answers count as wrong |
| `vitaldb-ioh@0` | VitalDB, numeric mean arterial pressure every 2 s | AUROC for hypotension (MAP < 65 mmHg for ≥ 60 s) in the 5 minutes after a 5-minute window; patient-level split |

Both are **preview** versions and may change. A result always names the exact task version and specification hash it was computed on.

## Datasets, licenses and citation

- **ClinicalQA**: CC BY 4.0 on KHDP (upstream Apache-2.0). Cite: SNUH-HARI. ClinicalQA: Korean Medical Knowledge QA Benchmark. Seoul National University Hospital Healthcare AI Research Institute, 2025. doi:10.23421/2026.0800. It has been public on Hugging Face since 2025-04-04; models trained after that may have seen it.
- **VitalDB**: CC BY 4.0. Cite: Lee HC, Park Y, Yoon SB, Yang SM, Park D, Jung CW. VitalDB, a high-fidelity multi-parameter vital signs database in surgical patients. Sci Data. 2022;9:279. doi:10.1038/s41597-022-01411-5.

medbench results and per-item predictions are released under CC BY 4.0. When you use a medbench result, cite the dataset and the result (BibTeX is in `/v1/results/{run_id}`).

## Tiers

- **baseline**: run by medbench with its own prompts or models. All p0 results are in this tier.
- **official**: run from code submitted by the model's authors (not available in p0).

## Not yet available

Submitting models, API tokens and the `medbench` CLI are planned. Until then, to request an evaluation, open an issue or contact the maintainers listed on https://medbench.kr/about.

## Errors

Errors are JSON with a machine-readable code: `{"error": "task_not_found", "message": "..."}`. Codes: `task_not_found`, `run_not_found`.

medbench.kr is an independent project using open SNUH datasets. It is not an official service of Seoul National University Hospital.