Qwen3-ASR 1.7B on k-or-speech-asr@0

Run a8444ec73f4e · standard · evaluated 2026-10-29

Scores

cer	0.073 (0.068–0.076)
wer	0.260 (0.250–0.270)
ser	0.464 (0.449–0.480)
parse rate	1.000
n	8000
tokens (in / out)	0 / 0
cost (USD)	5.3334

By voice

chul	0.043 (0.040–0.047) WER 0.213 (n=2000)
ethan	0.101 (0.095–0.107) WER 0.300 (n=2000)
maia	0.084 (0.078–0.089) WER 0.276 (n=2000)
sohee	0.062 (0.057–0.066) WER 0.251 (n=2000)

By code-switching style

english	0.206 (0.153–0.263) WER 0.347 (n=188)
mixed	0.110 (0.100–0.121) WER 0.398 (n=1604)
none	0.031 (0.028–0.035) WER 0.147 (n=2220)
phonetic_kr	0.070 (0.065–0.075) WER 0.252 (n=3988)

By intent

airway	0.093 (0.069–0.121) WER 0.296 (n=304)
bleeding_alert	0.051 (0.040–0.063) WER 0.216 (n=384)
chitchat	0.032 (0.020–0.051) WER 0.107 (n=360)
closure_status	0.076 (0.061–0.095) WER 0.270 (n=440)
confirmation	0.341 (0.341–0.341) WER 0.625 (n=4)
consultation_request	0.059 (0.043–0.078) WER 0.199 (n=232)
drug_administration	0.137 (0.120–0.154) WER 0.436 (n=552)
emergency	0.067 (0.053–0.084) WER 0.387 (n=304)
environment_request	0.063 (0.043–0.085) WER 0.228 (n=188)
equipment_issue	0.050 (0.038–0.064) WER 0.200 (n=272)
equipment_setting	0.048 (0.037–0.062) WER 0.212 (n=472)
handoff	0.066 (0.051–0.084) WER 0.254 (n=372)
imaging_request	0.074 (0.047–0.108) WER 0.249 (n=280)
instrument_request	0.071 (0.061–0.080) WER 0.221 (n=896)
neuromonitoring	0.088 (0.066–0.114) WER 0.306 (n=260)
patient_communication	0.025 (0.006–0.053) WER 0.104 (n=36)
specimen_handling	0.103 (0.075–0.134) WER 0.205 (n=364)
status_update	0.052 (0.044–0.061) WER 0.198 (n=964)
surgical_count	0.060 (0.047–0.076) WER 0.242 (n=276)
teaching	0.015 (0.010–0.021) WER 0.093 (n=260)
timeout_checklist	0.105 (0.105–0.105) WER 0.300 (n=4)
transfusion	0.068 (0.051–0.089) WER 0.344 (n=220)
vital_callout	0.145 (0.124–0.165) WER 0.503 (n=556)

By speaker role

anesthesia_attending	0.077 (0.064–0.090) WER 0.308 (n=800)
anesthesia_resident	0.080 (0.069–0.090) WER 0.289 (n=1244)
anesthesiologist	0.199 (0.059–0.365) WER 0.478 (n=12)
assistant	0.089 (0.072–0.107) WER 0.266 (n=500)
circulating_nurse	0.075 (0.063–0.090) WER 0.282 (n=836)
fellow	0.083 (0.069–0.097) WER 0.260 (n=860)
resident	0.088 (0.071–0.106) WER 0.290 (n=476)
scrub_nurse	0.063 (0.054–0.074) WER 0.245 (n=804)
surgeon	0.058 (0.052–0.064) WER 0.216 (n=2468)

By numbers in the reference

with digits	0.131 (0.122–0.141) WER 0.465 (n=1704)
without digits	0.055 (0.051–0.059) WER 0.195 (n=6296)

Reuse this result

Download per-item predictions (parquet) · JSON

@misc{medbench_a8444ec73f4e,
  title = {Qwen3-ASR 1.7B on Korean operating-room speech recognition (k-or-speech-asr@0)},
  author = {{medbench.kr}},
  year = {2026},
  howpublished = {\url{https://medbench.kr/r/a8444ec73f4e}},
  note = {Evaluated 2026-10-29, harness asr-p0.1, run key 73349543b164}
}

Versions

model	`alibaba/qwen3-asr-1.7b` (Alibaba)
task spec hash	`27b75c766635e4f4f96a4997cf25c8238f344a13fb93ec0c3332df53ca410fec`
harness	`asr-p0.1`
run key	`73349543b164d916ac44c2a973df1438786e918bce154c52fd4179223c4f9154`
config	`{"language": "ko", "response_format": "json", "temperature": 0}`

Data

Synthetic K-OR Speech Audio, Apache-2.0 with a use restriction.

Seoul National University Hospital, Department of Anesthesiology and Pain Medicine. Synthetic K-OR Speech Audio: A Synthetic Korean Operating Room Speech Audio Dataset (version 1.0.0). KHDP. 2026. doi:10.23421/2026.0774