Qwen3.8 Max 2.4T-A95B on kor-surgical-robot-voice-command@0

Run 4c3036eee37b · standard · evaluated 2026-10-29

Scores

exact_match	0.947 (0.916–0.974)
unsafe_execution	0.159 (0.028–0.313)
stop_recall	1.000 (0.794–1.000)
over_refusal	0.002 (0.000–0.007)
exact_match_asr_generic	0.754 (0.706–0.799)
unsafe_execution_asr_generic	0.220 (0.104–0.352)
exact_match_asr_kor_or	0.735 (0.692–0.779)
unsafe_execution_asr_kor_or	0.171 (0.049–0.304)
false_activation	0.045 (0.030–0.065)
parse rate	1.000
n	2085
tokens (in / out)	2213854 / 76847
cost (USD)	0.0000

Subset

clean	0.947 (n=495)
asr_generic	0.754 (n=495)
asr_kor_or	0.735 (n=495)
overheard	0.918 (n=600)

Type

clarify	1.000 (n=16)
explicit	0.990 (n=299)
implicit	0.741 (n=27)
indirect	1.000 (n=37)
reject	0.803 (n=66)
self_correction	0.912 (n=34)
stop	1.000 (n=16)

Form

abbr	0.963 (n=81)
abbr_ko	0.944 (n=71)
en	0.970 (n=133)
ko	0.939 (n=115)
natural	0.916 (n=95)

Reuse this result

Download per-item predictions (parquet) · JSON

@misc{medbench_4c3036eee37b,
  title = {Qwen3.8 Max 2.4T-A95B on Korean surgical robot voice commands (kor-surgical-robot-voice-command@0)},
  author = {{medbench.kr}},
  year = {2026},
  howpublished = {\url{https://medbench.kr/r/4c3036eee37b}},
  note = {Evaluated 2026-10-29, harness voice-command-p0, run key 8c7e1f62e3ce}
}

Versions

model	`alibaba/qwen3.8-max-2.4t-a95b` (Alibaba)
task spec hash	`3bc7dc79688a46dc00af2790c9d049d196d1da86de20a637020129b5336e6551`
harness	`voice-command-p0`
run key	`8c7e1f62e3ce505a5d51742fed28ebaa3d3af14a9dcc2382cc6a763df35b2644`
config	`{"max_tokens": 800, "prompt": "voice-command-v0.1", "response_format": "json_schema", "subsets": ["clean", "asr_generic", "asr_kor_or", "overheard"], "temperature": 0, "thinking": false}`

Data

Korean Surgical Robot Voice-Command Benchmark, CC BY 4.0.

Lee H. Korean Surgical Robot Voice-Command Benchmark (version 1.0.0). Seoul National University Hospital. KHDP. 2026. doi:10.23421/2026.03507