Run 9bba2b5b5329 · standard · evaluated 2026-10-29
Scores
| exact_match | 0.228 (0.177–0.281) |
|---|
| unsafe_execution | 0.988 (0.962–1.000) |
|---|
| stop_recall | 0.938 (0.812–1.000) |
|---|
| over_refusal | 0.000 (0.000–0.020) |
|---|
| exact_match_asr_generic | 0.137 (0.097–0.181) |
|---|
| unsafe_execution_asr_generic | 1.000 (0.912–1.000) |
|---|
| exact_match_asr_kor_or | 0.141 (0.101–0.187) |
|---|
| unsafe_execution_asr_kor_or | 1.000 (0.912–1.000) |
|---|
| false_activation | 0.998 (0.995–1.000) |
|---|
| parse rate | 0.999 |
|---|
| n | 2085 |
|---|
| tokens (in / out) | 2270284 / 103469 |
|---|
| cost (USD) | 0.0785 |
|---|
Subset
| clean | 0.228 (n=495) |
|---|
| asr_generic | 0.137 (n=495) |
|---|
| asr_kor_or | 0.141 (n=495) |
|---|
| overheard | 0.002 (n=600) |
|---|
Type
| clarify | 0.000 (n=16) |
|---|
| explicit | 0.254 (n=299) |
|---|
| implicit | 0.556 (n=27) |
|---|
| indirect | 0.243 (n=37) |
|---|
| reject | 0.000 (n=66) |
|---|
| self_correction | 0.029 (n=34) |
|---|
| stop | 0.750 (n=16) |
|---|
Form
| abbr | 0.272 (n=81) |
|---|
| abbr_ko | 0.042 (n=71) |
|---|
| en | 0.233 (n=133) |
|---|
| ko | 0.183 (n=115) |
|---|
| natural | 0.379 (n=95) |
|---|
Reuse this result
Download per-item predictions (parquet) · JSON
@misc{medbench_9bba2b5b5329,
title = {MedGemma 1.5 4B on Korean surgical robot voice commands (kor-surgical-robot-voice-command@0)},
author = {{medbench.kr}},
year = {2026},
howpublished = {\url{https://medbench.kr/r/9bba2b5b5329}},
note = {Evaluated 2026-10-29, harness voice-command-p0, run key eb83e73039ea}
}
Versions
| model | google/medgemma-1.5-4b (Google) |
|---|
| task spec hash | 3bc7dc79688a46dc00af2790c9d049d196d1da86de20a637020129b5336e6551 |
|---|
| harness | voice-command-p0 |
|---|
| run key | eb83e73039ea113f7439d69813f89762bff8ee8f074e5b5b8ceff3bd3d2a2c09 |
|---|
| config | {"max_tokens": 800, "prompt": "voice-command-v0.1", "response_format": "json_schema", "subsets": ["clean", "asr_generic", "asr_kor_or", "overheard"], "temperature": 0, "thinking": false} |
|---|
Data
Korean Surgical Robot Voice-Command Benchmark, CC BY 4.0.
Lee H. Korean Surgical Robot Voice-Command Benchmark (version 1.0.0). Seoul National University Hospital. KHDP. 2026. doi:10.23421/2026.03507