Clinical text › Synthetic SNUH (SYN-SNUH): OMOP CDM v5.4 synthetic EHR with LLM-generated Korean clinical notes › syn-snuh-chatqa@0

Multi-turn evidence-grounded QA over a patient's Korean notes preview

Data: doi.org/10.23421/2026.0770 · CC BY 4.0 · mcq_multiturn · primary metric: sample_acc_paired · spec d7b29f9d21fb

No results yet.

Data

Synthetic SNUH (SYN-SNUH): OMOP CDM v5.4 synthetic EHR with LLM-generated Korean clinical notes (text). License: CC BY 4.0. Source: https://doi.org/10.23421/2026.0770

Hyungchul. Synthetic SNUH (version 1.0.0). Seoul National University Hospital. KHDP. 2026. doi:10.23421/2026.0770

Task specification

{
  "baselines": {
    "medbench/prior": "always the dev split's most frequent correct letter, D: a tie, since A-D are each right on 3,095 dev turns (E on 3,094), and ties go to the later letter; sample-level paired 0.000",
    "medbench/random-letter": "uniform letters, numpy seed 0; sample-level paired 0.000"
  },
  "checks": {
    "evidence_from_structure": "with the questions, the earlier right answers and the choices shown so far (never the notes), the note the question names plus its conventional section is right on 6,493 of the 6,602 evidence turns whose wording names the note (date, 'the visit after the one on', set answer, 'that visit', 'those two visits', the most recent visit; 0.983), and 0.897 over all 7,785 (the review's probe on the build it reviewed: 0.779 overall, 0.954 where the question names the note). The evidence answer therefore follows from the right content answer on most turns, as in the paper's chartdate-anchored questions; it is hard only where the question names its visit by content or spans many notes (counts 0.32-0.45, highest or lowest value 0.335, next visit where measured 0.461, visit seen for a diagnosis 0.569).",
    "leak_probes": "notes-blind, on test (840 samples, 7,785 content and 7,785 evidence turns); trained probes are trained on dev. Content turns (chance 0.200): longest 0.196, shortest 0.206, most similar to the others 0.202, median value 0.104 (counts 0.18-0.24), TF-IDF char_wb 1-5 + logistic regression 0.276 (lab turns 0.21-0.24, diagnosis turns 0.12-0.34, drug turns 0.26-0.38: the dataset's most prescribed drugs fill only one of four distractor slots, so popularity remains a weak cue). Diagnosis from the drugs and labs the conversation has shown (the review's drug x code probe, with every earlier right answer): 0.175 (dx_at) and 0.210 (dx_next) with a table learned from dev, 0.180 and 0.210 with the release's own drug and lab counts per condition. Medication choice sharing the most or the fewest drugs with earlier right answers: 0.15-0.23 per turn type. Evidence turns: most similar 0.206, shortest 0.305, character probe 0.409, dates named in the questions 0.541, conventional section (drugs P>, labs O>, diagnosis #) 0.443, both 0.670. QA-level paired: random 0.039, character probe 0.116. Sample-level paired accuracy of every probe 0.000.",
    "pilot": "free KHDP-relayed models on 30 dev samples, 2 per category x note count (550 turns), these items: GLM-5.3 0.967 (0.900-1.000), Qwen3.8 Flash Next 0.733 (0.567-0.867), DeepSeek V4.1 Flash 0.667 (0.500-0.833), DeepSeek V4 Pro 0.600 (0.433-0.767), Qwen3.8 Max 0.433 (0.267-0.600; 59 of 550 replies have no letter the authors' parser reads, 0.700 re-read leniently). GLM-5.3 writes its reasoning into the reply despite the thinking-off fields (median 320 characters, ending '</think>B'); the others answer with the letter alone. Headroom against the prior is H = 0.033 for GLM-5.3, below the SOP's 0.05, and H = 0.27 for the best direct answer: the check fails for a model that reasons. Recounted from the saved replies, the four letter-only models answered 51 paired turns with a letter the authors' parser reads but a wrong content or evidence answer: visits with a lab at or above its threshold (lab_count) 17, change between two visits (lab_change) 11, highest or lowest value (lab_extreme) 6, next visit where the lab was measured (lab_next) 5, medication at the visit seen for a diagnosis (rx_by_dx) 4, visits with that medication (rx_count) 3, and one each of dx_count, dx_visits, rx_visits, dx_next and rx_at. All but the rx_at turn span visits or find a visit by its order or diagnosis. Qwen3.8 Max lost 51 further paired turns to replies its parser cannot read; GLM-5.3 missed one rx_at turn. Every count and set answer of both splits (1,696 turns) matches a recount from the raw note text. Earlier builds: CHANGELOG 2026-10-30."
  },
  "dataset": "syn-snuh",
  "decisions": [
    "Headroom: the SOP check fails on the pilot. GLM-5.3 reasons in its replies despite the thinking-off fields and scores 0.967 (0.900-1.000), H = 0.033 against the prior (the SOP asks 0.05); the letter-only models score 0.433-0.733. The code and these items are merged with the task hidden, and the task is not published on them.",
    "Next work, not built in this version: harder items, with more notes per sample and question types that need temporal reasoning across visits. They are built, probed and piloted before the task is published."
  ],
  "deviations": [
    "Protocol transfer: the authors' benchmark is MIMIC-IV discharge summaries with LLM-generated, expert-reviewed items; here their prompt, conversation, parser and scorer run unchanged at the pinned commit on items medbench.kr built from SYN-SNUH (owner decision 2026-10-30).",
    "Cohort: one window of 4, 8, 16, 24 or 32 consecutive outpatient notes per patient (distinct dates, all five sections, no Han characters or prompt text), not all of a patient's 1-5 discharge summaries; SYN-SNUH notes are about 160 characters, so 32 of them are about as long as one or two discharge summaries. Checked: tests/test_chatqa_items.py builds every cell on an invented release; the release fills all 15 cells with 56 patients in each split.",
    "Items: three of the eight categories (medication, diagnosis, clinical assessment), filled by rule from OMOP facts with the authors' templates as patterns; they seeded Gemini-2.5-Pro and had 12 medical experts review every sample. SYN-SNUH outpatient visits record no procedures, microbiology, outcomes or discharge plans, and symptom findings are MeSH terms the notes often write in Korean (91,971 of 238,067 not verbatim), so a distractor's absence could not be checked. 4-12 content turns, test mean 9.3 (the paper: 4-13, mean 8.3). No expert or LLM review. Checked on this build (both splits): every reference is unambiguous by construction (references above; tests/test_chatqa_items.py checks the wording on the built items); all 13,047 single-visit content answers (2,064 diagnoses, 6,713 medication sets, 4,270 lab values) are stated in their note; the evidence answer equals the builder's sources on all 15,522 evidence turns; every count and set answer (1,696 turns) matches a recount from the raw note text.",
    "Conversation rules: a visit's drug or lab names its diagnosis in SYN-SNUH, so the rules under items.conversation_rules keep earlier turns from giving a later answer away (they replace the build's drug-to-diagnosis questions, which a drug-to-code table answered at 0.89-0.94 in the review). Checked: tests/test_chatqa_items.py on the built items, and the conversation probes under checks (diagnosis from shown drugs and labs 0.175-0.210, chance 0.200).",
    "Fact rules: a fact is used only where the note states it and agrees with OMOP (items.gold). These checks are implemented here; the syn-snuh-note-errors clean filter is not merged, and of it only the accepted title names are reused (next deviation); its guideline-drug and plausibility rules judge clinical care, which this task does not ask about. Checked: tests/test_chatqa_items.py (title code and name, one section per drug, stopped and considered drugs, one stated value per lab under any label, window consistency for multi-note questions); on the release, 7,857 notes with a visit drug in a stop or consider clause give no medication fact.",
    "Diagnosis names: every question and choice shows one Korean name per code (DX_NAMES in chatqa_data.py: 당뇨병, 이상지질혈증, 우울증, 고혈압, 감기, 알레르기비염, 위염, 아토피피부염, 퇴행성관절염, 요통), a name of the note-errors accepted list; a diagnosis is a fact only when the note's own title name is in that accepted list for its code (ACCEPTED_DX_NAMES, the syn-snuh-note-errors design's HEADER_NAMES copied as is), so the shown name is the note's name or an accepted synonym of it. Checked: 12,670 of 162,042 usable notes whose title code matches OMOP (7.8%) have a title name outside the list (e.g. 비알레르기성 비염 (J30), 천포창 (L20), 족부통증 (M17), 요추디스크 (M54)) and give no diagnosis fact; tests/test_chatqa_items.py.",
    "Answer choices: built by rule from OMOP facts (items.answer_choices) instead of LLM brainstorming; numeric ranks, set draws and the exclusion of earlier answers are leak controls added here. Checked: the leak probes under checks.",
    "Evidence choices: the note's section markers (#, S>, O>, A>, P>) as headers instead of discharge-summary section names; distractors by the paper's error patterns, centred on a wrong variant with probability 0.8, never an earlier evidence answer. Checked: the leak probes and the structure probe under checks.",
    "Split: patients 50/50 by sha256 of a private salt, one sample per patient (the authors released one fixed set). Checked: 840 test and 840 dev patients, no overlap (tests/test_chatqa_items.py checks disjointness and one sample per patient).",
    "Letters: balanced A-E over each split and shuffled with the salt (the authors: balanced globally, seed 42). Checked: test 3,114 of each letter, dev 3,095/3,095/3,095/3,095/3,094.",
    "Note format: each note is prefixed with 'Chartdate: YYYY-MM-DD' inside the authors' [note #N start] wrapper (they prefixed an admission id and chartdate); no visit id is shown. Questions are English over Korean notes (the paper: English over English). Checked: tests/test_chatqa_protocol.py (wrapper).",
    "Request: thinking is switched off with each serving route's fields; the authors' API runner sent no such field and stripped <think> blocks (also done here). Checked: the pilot's replies left no open <think> block. The fields had no effect on GLM-5.3: in 547 of 550 pilot replies it reasons before '</think>' and then gives the letter (checks.pilot); the other four pilot models answer with the letter alone.",
    "Intervals: 95% bootstrap intervals over samples are added; the paper reports point estimates. Checked: tests/test_chatqa_eval.py."
  ],
  "instance_id": "sample id: the first 16 hex characters of sha256(salt, 'sample', person_id); one instance is one conversation",
  "items": {
    "answer_choices": "by rule from OMOP facts, never by an LLM. Drugs: up to two of the patient's own drugs for the same condition from other visits, then drugs the dataset prescribes for that condition, in proportion to how often; none stated in the note asked about; for several drugs, other drug sets the dataset prescribes together for that condition. Diagnoses: the patient's other diagnoses, then outpatient conditions in proportion to their frequency; none the note states (code, accepted name or common abbreviation). Values: same precision as the note, inside the dataset's range for the lab, the right value at a random rank among the five; the patient's other values first. Changes: the sign x magnitude grid. Visit sets: other sets of at most six dates of the same size, drawn at random. Counts: five consecutive counts, the right one at a uniformly drawn rank. None is an earlier answer of the conversation.",
    "categories": "medication, diagnosis, clinical assessment (labs and blood pressure); question chains adapted from the authors' templates of these categories, each repeated from new anchor visits while turns are left",
    "conversation_rules": "in SYN-SNUH 280 of 345 drugs (74% of prescriptions) belong to one condition and the rest mostly to one, and each lab to one (LDL, HDL, TG: E78; BP: I10; HbA1c, FPG: E11), so a drug or a lab names its visit's diagnosis. Hence: no question asks a diagnosis from a drug; a diagnosis is asked only at a visit no earlier turn has shown; a medication only at a visit no earlier medication turn has shown; a lab's highest or lowest value only before any other question on that lab; no answer or evidence set repeats an earlier one; no choice is an earlier answer, an earlier evidence answer or a diagnosis named in a question; a set question lists, in its answer and choices, only visits no earlier turn has shown.",
    "evidence_choices": "'Note #N Chartdate: YYYY-MM-DD Headers: ...' with the note's own section markers (#, S>, O>, A>, P>). The right set is the minimal set of (note, section) holding the answer (for a highest or lowest value or a count, every note it spans). Distractors follow the paper's error patterns (another section, another note, part of the set, the set plus an irrelevant source); with probability 0.8 the choices are built around a wrong one-edit variant, so the right set is not their centre; none is an earlier turn's evidence answer, and none contains a section that also states the asked diagnosis (e.g. 'A> 위염, well ctrl'd' beside the title '#위염 (K29)'), which a reader could defend as a minimal source too.",
    "gold": "from the OMOP rows of each visit (condition_occurrence, drug_exposure, measurement), and only for facts the note states verbatim and that agree with those rows: the title's ICD-10 code equals the visit's condition, its Korean name is one the syn-snuh-note-errors design accepts for the code, and no other section names the code; every OMOP drug of the visit is named in exactly one section, in no clause that stops, holds, replaces or only considers a drug (중단, 중지, 보류, 고려, 예정, 'X에서', stop, discontinue, d/c, hold, consider), and the note names no other dataset drug; a lab's value is stated once (also under other labels, Korean included), in one section, as the OMOP value truncated or rounded to the note's precision. Questions over several notes (which other visits, highest value, next visit where measured, the visit seen for a diagnosis) also require every note of the window to agree with OMOP on that drug, diagnosis or lab.",
    "letters": "balanced A-E over each split, shuffled with the salt (the authors balance letters globally with seed 42)",
    "question_kinds": "by visit (date, next visit, most recent visit, visit seen for a diagnosis): its medication(s), diagnosis or a lab value; across visits: the other visits where a medication or diagnosis recurs (at most six dates) or how many visits had it (the anchor visit included), the next visit where a lab was measured and the change between the two, a lab's highest or lowest value, how many visits had a lab at or above its clinical threshold (systolic BP 140 mmHg, HbA1c 7.0%, FPG 126, LDL 130, TG 150, HDL 40 mg/dL). Count choices are five consecutive counts with the right one at a uniformly drawn rank; where that would leave the possible range the question is not asked.",
    "references": "every question names its visit by date ('at the visit on 2023-04-23'), as 'the visit after the one on <date>', by a diagnosis no other note of the window has ('the visit where the patient was seen for X'), as 'the most recent visit in these notes', or by the turn just before it ('that visit', 'that later visit', 'the visit with that reading', 'those other visits' or 'that other visit' after a set question, 'the next visit where it was measured' straight after its anchor question). Highest and lowest values and counts take every note they span as their evidence.",
    "sample": "one patient, a window of k = 4, 8, 16, 24 or 32 consecutive outpatient notes (distinct dates, each with its title line and S>, O>, A>, P> sections), in date order",
    "turns": "4-12 content questions per sample (test mean 9.3; 390 of 840 samples have 12), each followed by the paper's evidence-grounding question"
  },
  "kind": "mcq_multiturn",
  "license": "CC BY 4.0 (SYN-SNUH); evaluation code MIT (EHRNote-ChatQA) within medbench.kr's Apache-2.0 export",
  "limitations": [
    "Synthetic data: no real patient records. Ten outpatient conditions, labs from truncated ranges, notes written by one generator (Qwen3.6-35B-A3B); its family may recognise its own phrasing.",
    "The notes are short and formulaic, so content questions are mostly lookup and linking across notes, not clinical reasoning; difficulty comes from long windows, set and extreme questions and the strict sample-level metric.",
    "The notes and their OMOP rows are public: anyone holding the release can answer the questions, and a model trained on SYN-SNUH may have seen them.",
    "The evidence answer follows from the right content answer on most turns (checks.evidence_from_structure).",
    "The conversation rules shape the chains: a diagnosis follows a drug or lab question only at another visit, so some of the authors' question kinds ('what was it prescribed for') are not asked.",
    "The source sometimes states clinically odd findings or drugs (measured in the syn-snuh-note-errors design); the questions ask what the notes record, not whether the care was right."
  ],
  "metrics": {
    "beside": [
      "qa_acc_content_individual",
      "qa_acc_source_individual",
      "qa_acc_paired",
      "sample_acc_content_individual",
      "sample_acc_source_individual",
      "sample_acc_content_individual_mean",
      "sample_acc_source_individual_mean",
      "sample_acc_paired_mean",
      "lenient_parser: the same replies re-read with the authors' Gemini parser where theirs finds no letter (a bare final letter after reasoning); shows what a model loses to its answer format, never the score"
    ],
    "breakdowns": [
      "category",
      "num_notes",
      "turn type"
    ],
    "intervals": "95% percentile bootstrap over samples (one per patient), 2,000 resamples, numpy seed 0",
    "primary": "sample_acc_paired",
    "sample_acc_paired": "Sample-level (0/1) Paired accuracy: a sample counts only if every content answer and its evidence answer are right (the paper's strictest metric)",
    "unparseable": "an answer without a letter is wrong (the authors' scorer); unparseable counts are reported"
  },
  "parser": "the authors' extract_answer_letter, unchanged: the default parser, or their DeepSeek-R1-Distill or Gemini variant chosen by model name as their scorer chooses it. The replayed history reads each reply as their API runner does (the chosen option in full, or the reply itself when no letter is found). An answer without a letter is wrong.",
  "preview": true,
  "primary_metric": "sample_acc_paired",
  "protocol": {
    "commit": "aa33511d43601ddca949d66a7b6f12c573640610",
    "license": "MIT",
    "medbench_made": [
      "cohort",
      "items",
      "conversation rules",
      "fact rules",
      "diagnosis names",
      "answer choices",
      "evidence choices",
      "split",
      "letters",
      "note format",
      "request",
      "intervals"
    ],
    "name": "EHRNote-ChatQA (Kim et al. 2026)",
    "note": "Protocol transfer (owner decision 2026-10-30): the authors built and ran their benchmark on MIMIC-IV discharge summaries, never on SYN-SNUH. Their prompt (byte for byte), note wrapper, multi-turn conversation and history format, letter parser (chosen by model name as their scorer does) and nine accuracies run unchanged at the pinned commit (src/medbench/tasks/chatqa_protocol.py). tests/test_chatqa_protocol.py checks the prompt, note wrapper, turn order, parsers and scorer against their files vendored in third_party/ehrnote-chatqa/ (evaluate_single_csv included). The conversation and history format come from evaluation/multiturn_gpt_evaluate_api.py and core/multiturn_gpt_evaluate_common.py, which are not vendored: the test checks the port against expectations written from them, and the reviews of this build compared the port with them by hand at the pinned commit. Every part medbench.kr made is a deviation below, with how it was checked.",
    "repository": "https://github.com/ji-youn-kim/EHRNote-ChatQA",
    "source": "Kim J, Yeo M, Jang E, et al. EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries. arXiv:2606.15735 (NeurIPS 2026 Evaluations & Datasets)",
    "transfer": true
  },
  "request": {
    "max_tokens": 2048,
    "prompt": "prompt/multiturn_gpt_evaluate.txt (verbatim); the notes and the first question in the first user message, later messages only the next question; past turns replayed as the question without choices and the model's choice as '{letter}: {choice}'",
    "temperature": 0,
    "thinking": "off, with each serving route's request.thinking_off fields; any <think> block is stripped before parsing and before the reply enters the history (the authors' rule)"
  },
  "slug": "syn-snuh-chatqa",
  "source": {
    "files": "synthetic_snuh.duckdb, SHA-256 fd2056b31cc74c171c3906a7e48482979b993d7a52471dbe5ddd3f3df21a87c8",
    "khdp_code": "SYN-SNUH",
    "note_text": "note_llm_log.text (the LLM-written note); note.note_text is a template rendered from the rows and is not used",
    "version": "1.0.0"
  },
  "split": "test: 840 samples (15,570 turns) from 840 patients, 56 per category x note count; dev: 840 samples (15,474 turns) from 840 other patients (leak probes and pilots only). Patients are split 50/50 by a private salt; one sample per patient.",
  "title": "Multi-turn evidence-grounded QA over a patient's Korean notes",
  "version": 0,
  "visibility": "hidden"
}

JSON: /v1/tasks/syn-snuh-chatqa · /v1/leaderboard?task=syn-snuh-chatqa@0