DeepSeek V4.1 Flash 552B-A16B on synthetic-snuh-text2sql@0

Run ed9864699f04 · standard · evaluated 2026-10-30

Scores

hard_ex	0.596 (0.500–0.693)
rs0	0.663 (0.579–0.743)
hard_rs5	-1.020 (-1.525–-0.544)
hard_rsN	-67.337 (-84.421–-51.232)
easy_ex	0.994 (0.984–1.000)
parse rate	0.951
n	510
tokens (in / out)	2457440 / 1586668
cost (USD)	0.0000

Reuse this result

Download per-item predictions (parquet) · JSON

@misc{medbench_ed9864699f04,
  title = {DeepSeek V4.1 Flash 552B-A16B on Synthetic-SNUH Text-to-SQL (OMOP CDM, Korean and English) (synthetic-snuh-text2sql@0)},
  author = {{medbench.kr}},
  year = {2026},
  howpublished = {\url{https://medbench.kr/r/ed9864699f04}},
  note = {Evaluated 2026-10-30, harness text2sql-p0, run key 3115b9abc418}
}

Versions

model	`deepseek/deepseek-v4.1-flash-552b-a16b` (DeepSeek)
task spec hash	`abb0fa4ae8e35e706e34c4862ff8a1c1a3aba1f57c39900eb3ee3d6685f4650c`
harness	`text2sql-p0`
run key	`3115b9abc418765dd7741997778312e9a8bd77f1fff65a23cdf5976f4a399442`
config	`{"max_tokens": 16384, "prompts": {"easy": "text2sql-omop-p0", "hard": "text2sql-omop-hard-p0"}, "temperature": 0, "tiers": ["hard", "easy"]}`

Data

Synthetic-SNUH v4, CC BY-NC 4.0.

Lee H-C, Seoul National University Hospital. Synthetic-SNUH v4.0.0: synthetic OMOP CDM and KR Core FHIR records for 10,000 virtual patients (version 4.0.0). KHDP. 2026. doi:10.23421/2026.03183