About

When a new medical AI model appears, many groups evaluate it on the same public datasets, each paying for the same compute and each with slightly different task definitions. medbench.kr runs every model on every task once, under a fixed and versioned definition, and publishes the result so that others can cite it, download the predictions and build on them instead of repeating the run. Over time, models submitted by their own authors and scored under identical conditions will form an official leaderboard.

How results are produced

• Each task has a versioned specification; its hash is shown on the task page. Tasks marked preview may still change.
• Each result is identified by a run key: a hash of the model version, the task specification, the harness version and the run configuration. The same key is never computed twice.
• LLMs are queried with a fixed prompt at temperature 0. Answers that cannot be parsed count as wrong; the parse rate is reported.
• Confidence intervals are 95% bootstrap intervals: over questions for ClinicalQA, over patients for VitalDB.
• All current results are in the baseline tier: medbench ran them with its own prompts or models. Author-submitted results will form a separate official tier.

Reuse

Every result page links per-item predictions and a BibTeX entry. Agents should call /v1/lookup before running an evaluation themselves; each lookup that finds an existing result is counted as a reuse.

Data

All datasets are open datasets from Seoul National University Hospital, distributed through KHDP (khdp.net). Each task page credits its dataset and links its license.

Privacy

Visits are counted without cookies. An IP address is combined with a secret and the date and hashed; the address itself is never stored, and the hash changes every day. Each page also sends one small request saying which page was opened, so that real browser visits can be told apart from automated fetches.

Disclaimer

medbench.kr is an independent project using open SNUH datasets. It is not an official service of Seoul National University Hospital. Scores describe model behavior on these datasets only and are not clinical validation.