Clinical text › Synthetic SNUH (SYN-SNUH): OMOP CDM v5.4 synthetic EHR with LLM-generated Korean clinical notes › syn-snuh-note-errors@0

Clinical error detection in synthetic Korean SNUH notes (paired) preview

Data: doi.org/10.23421/2026.0770 · CC BY 4.0 · error_pair · primary metric: bcr · spec 7512d194f615

No results yet.

Data

Synthetic SNUH (SYN-SNUH): OMOP CDM v5.4 synthetic EHR with LLM-generated Korean clinical notes (text). License: CC BY 4.0. Source: https://doi.org/10.23421/2026.0770

Hyungchul. Synthetic SNUH (version 1.0.0). Seoul National University Hospital. KHDP. 2026. doi:10.23421/2026.0770

Task specification

{
  "baselines": {
    "medbench/constant-yes": "constant Yes (sanity row: BCR 0 by construction, F1 0.667)",
    "medbench/note-errors-conjunction-lr": "SOP A1.4's simple baseline that reads the input: word 1-2 grams and each word conjoined with the note's own header code, logistic regression (C 10) trained on the train pairs, Yes above the train-median score (the report-only conjunction probe as a recorded row)",
    "medbench/prior": "constant No (sanity row: BCR 0 by construction)",
    "medbench/random": "Yes with probability 0.5 on each note, numpy default_rng(0); its expectation, 0.25, is the input-blind reference"
  },
  "check": {
    "beats_prior": "the weak reference's BCR CI lower bound is above 0.25, and so is every baseline's that reads the input (medbench/note-errors-conjunction-lr); constant rows are sanity rows, and random's CI covers 0.25",
    "effective_ceiling": "0.951 = 1 - 0.049: the clean audit's share of twins flagged by two or more reviewers, weighted by rule size (5 of 100 twins on the pre-exclusion set; design 20.5); the marked-pair check found no pair of a kept rule whose error two reviewers called unreal or undetectable (invalid-error rate 0). The audit sampled twins before the review of PR #107 removed wrong-indication twins that contradicted their note (design 21.2), so it is an estimate, not a bound",
    "format": "parse failures below 10% per roster model; API errors of 10% or more make the check incomplete",
    "generator": "alibaba/qwen3.6-35b-a3b, run once, tagged generator, never a reference; Qwen models are not references",
    "headroom": "H = (C - best) / (C - 0.25) >= 0.05 with the best reference's CI below C, for C = 1.0 and for the effective ceiling from the three-model item check (residual slip and invalid-error rates); best excludes the generator, and the best non-Qwen model is reported too",
    "other_reference": "zhipu/glm-5.3-753b-a40b",
    "prior": 0.25,
    "separation": "the strong and weak references' CIs do not overlap, strong above; the paired bootstrap of the BCR difference over pairs (2,000 resamples, numpy default_rng(0)) is reported with it",
    "strong_reference": "deepseek/deepseek-v4-pro-1.6t-a49b",
    "thinking": "a roster run counts as thinking-off only when its thinking audit holds (no reply with a reasoning field or a think block); otherwise it is reported as reasoning-on and not compared as a thinking-off reference. The weak reference is the exception (decisions, 2026-10-31): gpt-oss-20b cannot switch reasoning off, runs at the lowest effort its route offers (reasoning_effort low) and is compared as a reasoning-on weak reference",
    "weak_reference": "openai/gpt-oss-20b"
  },
  "counts": {
    "test": {
      "per_category": {
        "diagnosis": 160,
        "lab": 160,
        "pharmacotherapy": 300
      },
      "per_condition": {
        "E11": 77,
        "E78": 10,
        "F32": 91,
        "I10": 87,
        "J30": 45,
        "K29": 64,
        "M17": 149,
        "M54": 97
      },
      "per_edited_line": {
        "A": 240,
        "O": 80,
        "P": 300
      },
      "per_rule": {
        "DX05": 80,
        "DX06": 80,
        "LI01": 15,
        "LI02": 18,
        "LI03": 15,
        "LI04": 22,
        "LI05": 10,
        "LI06": 25,
        "LI07": 25,
        "LI08": 15,
        "LI09": 15,
        "PH08": 64,
        "PH09": 86,
        "PH12": 14,
        "PH13": 91,
        "PH14": 45
      }
    },
    "train": {
      "per_category": {
        "diagnosis": 160,
        "lab": 160,
        "pharmacotherapy": 300
      },
      "per_rule": {
        "DX05": 80,
        "DX06": 80,
        "LI01": 15,
        "LI02": 18,
        "LI03": 15,
        "LI04": 22,
        "LI05": 10,
        "LI06": 25,
        "LI07": 25,
        "LI08": 15,
        "LI09": 15,
        "PH08": 66,
        "PH09": 84,
        "PH12": 22,
        "PH13": 83,
        "PH14": 45
      }
    }
  },
  "dataset": "syn-snuh",
  "decisions": [
    {
      "by": "lead (benchmark builder workflow), open questions of design section 15",
      "code_publication": "The rule and selection code and the split salt stay private while a test version of this task is live and are published when that version is retired. Prompts, parser, metrics and the scorer are public with the task.",
      "date": "2026-10-30",
      "paid_roster_runs": "Yes. google/medgemma-1.5-4b (weak reference) and alibaba/qwen3.6-35b-a3b (the generator, tagged, never a reference) run on the compute account's credits under the owner's budget for it, not under the $5 benchmark check cap.",
      "provisional_status": "Decided by the owner at publication; the task stays hidden until then."
    },
    {
      "by": "lead, ruling on the leak-gate result (design 18.10)",
      "categories": "Accept the four-category set the leak gate produced: 27 rules, 820 test and 820 train pairs (pharmacotherapy 300, dose 200, lab 160, diagnosis 160). G1-G4 pass. Management (MG02, MG03) is dropped because its plan-verb wording cannot pass G2 and G3 together in this release; the MEDEC management category is not covered (limitations), and its measurements are kept in the design.",
      "date": "2026-10-30",
      "fine_tuned_models": "A model trained on the train split could learn these associations, so any model fine-tuned on it is tagged and kept off the headline.",
      "report_only_probes": "The report-only probes stay report-only and are published in the limitations with their numbers. The design's bound of 30% of test pairs in rules at or above 0.90 on the conjunction or novelty probe is waived: these probes encode the clinical knowledge each item tests (a drug that does not fit the condition, a dose above the label maximum), so tuning pairs against them would remove the construct."
    },
    {
      "by": "three-model item check (design 10 and 20), applied by the builder",
      "clean_filter": "Patterns the clean audit flagged in source notes became clean-filter checks, and every visit whose twin a reviewer flagged is excluded from the pairs.",
      "date": "2026-10-30",
      "rules": "The rule table, marked-pair, blind and clean-audit verdicts were applied as design 10 decides them (rules above); the dose category is no longer covered. PH12 stays with one reviewer's objection open after two revisions, because without it no selection passed the surface gate G2 (0 of 38); the lead decides whether it stays (design 20.9)."
    },
    {
      "by": "lead (benchmark builder workflow), weak reference of the SOP check (design 22)",
      "date": "2026-10-31",
      "weak_reference": "openai/gpt-oss-20b replaces google/medgemma-1.5-4b as the weak reference. MedGemma 1.5 4B, fixed before the runs, answered llm_unavailable on its serving route on every probe from 02:15 to 03:32 KST on 2026-10-31 (four probes), so beats-the-prior and separation could not be computed. gpt-oss-20b (21B, 3.6B active) is on the same paid route under the same approval (paid_roster_runs) and was chosen before it saw any note of this task. It cannot switch reasoning off: it runs with its route's fields (reasoning_effort low, the lowest), so its thinking audit does not hold and the run is reported as reasoning-on. It is compared anyway, because reasoning can only make a weak reference stronger: separation from it is the harder test. Beats-the-prior also requires the input-reading baseline above 0.25, which does not reason."
    }
  ],
  "deviations": [
    "Items are built, not written: deterministic rules inject one substitution error into a note generated from structured data, and the clean twin is also edited (a clinically equivalent substitution at the same span) so that neither note is verbatim in the open release. The paper used clinician- or annotator-written errors with unedited clean notes.",
    "Error types: pharmacotherapy (MEDEC's pharmacotherapy type, wrong-indication swaps only), lab value interpretation (one of the MedErrBench-EN types) and diagnosis; no dose, management, treatment-procedure or causal-organism errors.",
    "Korean notes (with English abbreviations) inside the English MedErrBench-EN prompt, unchanged; the paper used native-language prompts for Chinese and Japanese.",
    "Protocol transfer: the authors never ran this dataset. Their prompts, metrics and ECA are ported; the parser is theirs with three additions (parser); the data construction is medbench's.",
    "The headline is one configuration (neutral, greedy); the paper's primary is the mean over four configurations, reported for the reference roster.",
    "max_tokens 2048 instead of 512 new tokens (GLM-5.3: 133 of 192 pilot replies unparseable at 512).",
    "A pair with a member without a verdict counts as not both-correct. The paper's text says such pairs are skipped and its table code imputes them at random (asymmetrically, fitted on the run itself); skip, the code's imputation and a corrected imputation are all reported.",
    "95% bootstrap CI over pairs; the paper reports the SD over configurations.",
    "Parser: the tolerant parser of the authors' MS-Test API runner (their MedErrBench-EN runs used the strict one, reported beside the headline), plus think-block removal, truncation without a closed answer as no verdict and the template echo as no verdict.",
    "ECA reference text is the note line holding the edit (the A line also counts for LI06-LI09), as the notes have no sentence-level error annotation; the code's categories and TP localisation are kept; flag localisation, span hit and the P-line baseline are medbench additions.",
    "Thinking is switched off with each serving route's fields and validated per-model overrides; the paper loaded weights locally with enable_thinking false where the chat template supported it. GLM-5.3 reasons with every switch its route documents, so its messages end with an empty think block that the reply continues (request); a run counts as thinking-off only when its thinking audit holds.",
    "Exact pairing by construction and one pair per patient (no Jaccard matching, no shared clean notes).",
    "Filter development used notes of patients that an earlier split had put in test (the pilot models' false positives on 105 clean twins); remedy: a new private split, and the 219 development patients are in train only.",
    "SOP: the check criteria are mapped to a paired task (check), and the three-model item check departs from the SOP item review (design 10: sampled pairs, unanimity per rule, no LLM fixes). The roster was fixed before the runs, but its weak reference was replaced after the other runs, because MedGemma 1.5 4B was unreachable; the replacement, gpt-oss-20b, reasons at low effort (decisions, 2026-10-31).",
    "PH12 is kept without the unanimity design 10 requires: after three rounds one reviewer still objected to one of two sampled twins (\"empagliflozin 증량\", a dose increase of a drug at its lower strength), while the other two approved every PH12 pair; without PH12 no selection passed G2 (0 of 38, design 20.9). 14 test pairs.",
    "The item-check reviewers saw 87 visits of the test pairs (item_check). The SOP marks reviewers if a benchmark ever ranks them; here models of their families are tagged and also scored without those pairs.",
    "Item check as run: the three reviewers have mandatory reasoning on the route, so they ran at the lowest effort (low) instead of with thinking off, and GPT and Claude ignore temperature; the marked-pair step sampled 2 pairs of each of 27 rules (design: 40 rules), and re-checks of revised rules sampled new pairs."
  ],
  "instances": {
    "files": "pairs_test.jsonl and pairs_train.jsonl, private (git-ignored data/ dir); the run commands refuse other files",
    "id": "<pair_id>/error and <pair_id>/clean; pair_id = sha256(salt, 'pair', visit)[:16] with the private salt",
    "sha256": {
      "test": "a297aa3978f3a16f2df832cb676b2149ec44365f4287b1d1f8edd67b7c00e064",
      "train": "6044d84bc35215a24c7939c6c5b0a8366c0971bbce7e15f441380d9104cd79ad"
    }
  },
  "item_check": {
    "A_rule_table": "27 rules and the tables; two of three decide. Revised: PH12, PH14, DO01-DO10, DX06; dose maxima of aceclofenac, eperisone, gliclazide, itopride and pseudoephedrine changed; every other rule and entry kept",
    "B2_blind": "30 pairs, notes in hashed order: GPT and Claude named the error note in 30 of 30, Gemini in 29 (one both)",
    "B_marked_pairs": "2 pairs per rule (54), unanimity per rule. Kept: PH08, PH09, PH13, PH14, LI01-LI07, LI09, DX05, DX06, DO01, DO03, DO04 (17). Revised and re-checked: LI08 (kept after re-check with LI09, all yes), PH12 (one reviewer still objected after two revisions; kept without unanimity, deviations). Revised but not re-checked, then dropped with the dose family: DO02, DO05-DO08, DO10. Dropped: PH11, DO09 (two or three no on a pair); the dose family as a whole (rules_dropped)",
    "C_clean_audit": "100 twins of the 16 kept rules (PH12 and the E11 and E78 lab rules oversampled; the dose rules were already gone): 17 flagged by at least one reviewer (weighted 0.165), 5 by two or more (weighted 0.049); the 17 and two twins flagged in step B are excluded with their pairs",
    "exposure": "The three reviewers saw 189 visits in steps B, B2 and C, 87 of them in the current test pairs (18 with the error marked in B and its re-checks, 20 blind in B2, 49 as twins in C). A model of their families (GPT-6, Gemini 4, Claude Opus 5) is tagged item-check-reviewer and its BCR is also reported without those pairs (design 21.6)",
    "spend_usd": {
      "A_rule_table": 0.6,
      "B2_blind": 0.335,
      "B_marked_pairs": 1.03,
      "B_rechecks": 0.212,
      "C_clean_audit": 0.913,
      "total": 3.091
    }
  },
  "items": {
    "categories": {
      "diagnosis": "the A-line diagnosis swapped between knee osteoarthritis and low back pain while the note keeps its knee or back cues; inserted knee forms are knee-specific (DX05, DX06)",
      "lab": "a control statement that contradicts the stated values (LI01-LI05), or values that contradict the control statement (LI06-LI09)",
      "pharmacotherapy": "a drug that does not fit the active condition (wrong indication): PH08, PH09, PH12, PH13, PH14. Since the review of PR #107 (design 21.2) a candidate is skipped when the note names a drug class or the original drug outside the edit, when a dose, frequency, quantity or route follows the drug in its clause, or when that clause stops, switches or changes it (the twin would contradict the note, or the error side would be correct care); PH08, PH13 and PH14 need the disease active in S and A as PH09 did, and PH08 skips dyspepsia assessments (antidepressants treat functional dyspepsia)"
    },
    "clean_filter": "Design section 4 with the build changes of section 18.3: generator slips (header code, accepted header names, drugs and labs agree with the visit's rows, stated doses within the label table), source problems (off-guideline drugs, duplicate classes, blocklisted HSDN findings), and assessments or plans that contradict the labs (control statements against HbA1c, FPG, BP and LDL; clinical inertia by lab value). The three-model item check (design 20) added: diabetic HbA1c and FPG labelled WNL or 정상, drug intensification at target (E11, I10), \"교육 강화\" no longer read as intensification, \"controlled\" with persisting symptoms, glucosamine as the only knee-OA drug; and its rule-table review raised the dose maxima of aceclofenac, eperisone, gliclazide, itopride and pseudoephedrine. 45,103 outpatient notes from 8,672 patients pass; ED notes are not used. 19 visits whose twin a reviewer flagged are excluded from the pairs.",
    "input": "one note inside the protocol prompt; nothing else (no structured rows, no twin, no age or sex)",
    "pair": "Both notes of a pair start from the same outpatient visit's note and are edited at the same span(s): the error note with one injected clinical error, the clean twin with a clinically equivalent replacement drawn from the same option pool. Neither note is in the release (gate G1). One pair per patient across both splits.",
    "rules_dropped": {
      "DO01-DO05": "surface gate: without the under-dose mirrors the over-dose rules cannot be balanced (G2 dose family 0.95)",
      "DO06, DO07, DO08, DO10": "check A and B: an under-dose is an error only for an adult and the notes give no age; one reviewer rejected all under-dose rules in the rule-table review and three of eight sampled under-dose errors as plausible paediatric doses, which no edit inside a note can resolve",
      "DO09": "check B: all three flagged both sampled twins (metformin started at 1000 mg bid)",
      "DX01": "G2 family 1.000 and G3 1.000: HTN is inserted only on the error side",
      "DX07, DX08": "G2 family 1.000: no second LDL form occurs in 20 or more clean notes",
      "MG02, MG03": "management (plan verbs): G2 0.733 for the category and G3 0.73-0.77 per rule; maintain and intensify forms cannot share one pool within G3 <= 0.75 in this release (lead ruling 2026-10-30, see decisions)",
      "PH01": "G3 1.000: the twin non-statin is always nearer the original than any statin",
      "PH02-PH07": "duplicate-class rules: with them pharmacotherapy G2 is 0.633 and the overall gate fails",
      "PH10": "G3 1.000: statin twins share -statin with the original",
      "PH11": "check B: all three reviewers flagged a twin (amlodipine for hypertension with albuminuria, where an ACE inhibitor or ARB is indicated)"
    }
  },
  "kind": "error_pair",
  "leak_gates": {
    "G1_lookup": "0 of 1,240 error notes and 0 of 1,240 twins equal a release note (pass)",
    "G2_char_wb_1_5_lr": "paired accuracy 0.473 (0.432-0.511), null 0.501, 97.5th percentile 0.540; BCR 0.247 (0.213-0.281), null 0.253; BCR minus null, 97.5th percentile 0.036; per category diagnosis 0.500, lab 0.506, pharmacotherapy 0.440 (pass). Fragile: 1 of 24 selections tried after the review of PR #107 passes. 1 of 18 selection rounds of this rule set passes (round 0 is kept); the others reach 0.510-0.545 with BCR minus null 0.044-0.084. None of 6 earlier selections passes; they used the rule set before the PH08 dyspepsia skip, with the code's weights and five weight variants, and reach 0.516-0.542 with BCR minus null 0.044-0.085. The robustness probes below read the same pairs at up to 0.718",
    "G3_source_distance": "0.501 (0.465-0.539); highest per rule PH12 0.607 (14 pairs; DX05 0.606) against 0.75 (pass). Whole-note check 0.550 (0.513-0.585); per rule DX05 0.756, PH09 0.622, PH13 0.637 and PH14 0.289 (0.711 read the other way)",
    "G4_counts_lr": "0.500: the train count differences balance exactly, so the probe has no weights and every pair ties (pass by construction, not a measurement); with no training, \"the longer note is the error\" gives 0.500 (0.465-0.534)",
    "release_holder": {
      "numeric_distance_to_release_values": "the side whose values lie farther from the release note's is the error: value rules LI06-LI09 0.925 (0.863-0.975; LI06, LI08 and LI09 1.000, LI07 0.760); 0.555 over all pairs (ties where no number is edited)",
      "patients_other_visits": "the side whose inserted drug or diagnosis appears in the patient's other visits is the twin: 0.608 (0.586-0.629); diagnosis 0.722, pharmacotherapy 0.605 (PH09 0.773, PH08 0.406); the twin's item is in the history in 274 pairs, the error's in 140, of 460 pairs with a drug or diagnosis edit"
    },
    "report_only": {
      "cased_char_1_8_lr": "0.618 (0.581-0.656); BCR 0.377, null 0.303",
      "corpus_novelty_10_features_lr": "0.706 (0.677-0.735)",
      "corpus_novelty_unseen_word_4grams": "0.681 (0.657-0.704), within the design bound of 0.75",
      "knowledge_or_corpus_signal": "The conjunction probe reads drug x condition and diagnosis x header code, the clinical judgement each wrong-indication and diagnosis item tests; the design's 30% bound is waived (decisions). Rules at or above 0.90 on the conjunction or a novelty probe hold 82% of the test pairs (510 of 620); with the rules read as strongly in reverse (at or below 0.10: LI01 0.067 and LI03 0.000 on the conjunction probe), 87% (540).",
      "qwen3_0_6b_log_likelihood": "0.552 (0.513-0.590); the lower-likelihood note called the error (cached weights)",
      "word_1_2_header_code_conjunctions_lr": "0.888 (0.863-0.912); BCR 0.773, null 0.414",
      "word_1_3_lr": "paired accuracy 0.751 (0.715-0.786); BCR 0.502, null 0.308"
    },
    "robustness": {
      "G2_trained_on_test_scored_on_train": "0.485 (0.447-0.526); BCR minus null, 97.5th percentile 0.050",
      "char_wb_1_5_presence_lr": "the G2 analyzer with binary features (no IDF, no normalisation): C 1 0.710 (0.673-0.745), BCR 0.324 against a null of 0.196 (BCR minus null, 97.5th 0.171), pharmacotherapy 0.810, diagnosis 0.644, lab 0.588; C 0.01 to 10: 0.698-0.718; trained on test and scored on train 0.681 (0.644-0.716)",
      "counts_x_header_code_lr": "0.556 (0.520-0.593); pharmacotherapy 0.677, diagnosis 0.362; BCR minus null, 97.5th 0.015",
      "g2_features_paired_loss": "C 0.1 0.466, C 1 0.484, C 10 0.606 (0.568-0.644), pharmacotherapy 0.657 at C 10"
    }
  },
  "license": "CC BY 4.0 (SYN-SNUH); evaluation code MIT (paired-clinical-eval, third_party/paired-clinical-eval) within medbench's Apache-2.0 code",
  "limitations": [
    "Synthetic data: no SNUH patient records. Conditions are 14 ICD-10 codes (8 in the test pairs), labs come from truncated ranges, HbA1c and FPG are drawn independently, and the notes are short (median 160 characters) and formulaic.",
    "The MEDEC management category is not covered: no plan-verb error (intensifying at target, clinical inertia at a high value) passes the surface-leak gates in this release (design 18.6); ED notes are not used.",
    "Dose errors are not covered (design 20.3): under-dose errors need the patient's age, which the notes do not give, and over-dose errors alone cannot pass the surface gate. Pharmacotherapy items are wrong-indication swaps only.",
    "The surface gate G2 passes in 1 of 24 selections tried after the review of PR #107 (1 of 18 rounds of this rule set, none of 6 with the rule set before the PH08 dyspepsia skip and five weight variants; before the review: 1 of 11, and none of 38 without PH12): the bounds are met by this selection, not by the construction in general. The same analyzer with presence features reads the pairs at 0.710 (0.673-0.745; pharmacotherapy 0.810), G2's features with a paired loss at up to 0.606, and counts conjoined with the header code at 0.556 (leak_gates.robustness). Some of this may be drug-by-condition association, which the items test; none of it is bounded.",
    "The notes are public. Twin edits keep test notes out of the release, but a reader holding the release can find each pair's source note, and G3 (0.501) bounds only the character distance of the edited span, not what that buys: the side whose values lie farther from the release note's is the error in 0.925 of the value-rule pairs (LI06-LI09), the side whose drug or diagnosis the patient has at other visits is the twin in 0.608 of all pairs (diagnosis 0.722, pharmacotherapy 0.605), and the whole-note distance reaches 0.756 on DX05 and 0.711 on PH14 read the other way (leak_gates.release_holder). These need no training and no clinical knowledge; they feed the contamination-check redesign (design 21.3).",
    "Knowledge or corpus signal (report-only probes, test split): word 1-3 0.751, word 1-2 with header-code conjunctions 0.888, cased char 1-8 0.618, corpus novelty 0.681 and Qwen3-0.6B log-likelihood 0.552 paired accuracy; rules at or above 0.90 on a conjunction or novelty probe hold 82% of the test pairs (510 of 620), 87% (540) with the rules read as strongly in reverse (LI01, LI03). These probes read the clinical judgement each item tests (a drug that does not fit the condition, a diagnosis that does not fit the header code), so they were not tuned against. A model trained on the train split could learn these associations: any model fine-tuned on it is tagged and kept off the headline, and the train split stays private.",
    "One generator (Qwen3.6-35B-A3B) wrote every note; its family, and any model trained on SYN-SNUH, may recognise its phrasing. The generator run is tagged and Qwen models are not references.",
    "The clean side is filtered, not verified: 45,103 of 177,221 outpatient notes pass, with guideline drugs only. The review of PR #107 found 12 clear and 5 arguable test twins whose edit contradicted the rest of the note (a drug class named elsewhere, the source's dosing kept, a stop plan); the rules now skip such notes (design 21.2), but no reviewer audited the rebuilt twins. In 26 of 160 diagnosis error notes of the pre-review set the header code was the only conflicting cue; this is unchanged. The three-model clean audit flagged 17 of 100 twins (5 by two or more reviewers); those visits are excluded and the patterns behind them filtered, but unaudited twins can still hold such slips. No clinician reviewed them. Twins keep the source's plan verb, so a PH twin can read as clinical inertia: 94 of the 300 test PH twins swap to a lower-step drug (a PPI to an H2RA or rebamipide, a nasal steroid to an oral antihistamine, an NSAID to acetaminophen; 86 of them with \"유지\") in active disease. The strong reference flags 6.4% of these against 9.2% of the other PH twins, and 8 of the 45 PH14 twins. The active-disease filter also misses stable wording it has no pattern for: one PH08 test twin keeps famotidine for \"Chronic gastritis, well maintained\" (its error side is still a wrong indication).",
    "Error types are substitutions of one or two spans; no omissions or insertions, and no errors that need age, sex, renal function or history.",
    "The rules' clinical judgements come from Korean guidelines and labels as written in the design, reviewed by three frontier models, not by a clinician panel.",
    "Any change to the rules needs a new selection and gate run; once published, a new task version."
  ],
  "metrics": {
    "bcr": "share of test pairs with Error: Yes on the error note and Error: No on the clean twin; a pair with a member without a verdict is not both-correct",
    "breakdowns": [
      "category",
      "rule",
      "condition",
      "edited_line"
    ],
    "evidence": "ECA over pairs flagged on both sides (categories A-E, X = no Evidence on either side) with TP localisation and FP correction as the code defines them; flag localisation (flagged error notes whose Evidence matches the edited line), span hit (Evidence contains the inserted text) and the P-line baseline (flag localisation of always quoting the P line: 0.484 on the test pairs), medbench additions. The reference line is the line holding the edit; for LI06-LI09 the A line also counts. Matching is the code's is_match (Hangul takes its word path); character coverage is reported as a check.",
    "intervals": "95% percentile bootstrap over pairs, 2,000 resamples, numpy default_rng(0)",
    "primary": "bcr",
    "reported": "BCR with the strict parser; BCR skipping pairs without a verdict; BCR with the code's random-assignment imputation (100 draws, Python random seeded 42) and with the corrected imputation; sensitivity, specificity, balanced accuracy, F1, precision, MCC, flag rate and compute_metrics (with per-category accuracy and recall on error notes); pair outcomes (both correct, both Yes, both No, both wrong); expected BCR under independence (sens x spec) and the independence ratio; parse failures by side, truncated replies and API errors"
  },
  "parser": "Drop every <think>...</think> block and keep only the text after the last </think>; a last Error line that echoes the template (\"No or Yes\") is no verdict; a reply cut at max_tokens is no verdict unless it closed a reasoning block and wrote an Error line after it; otherwise the authors' tolerant parse_error_verdict (run_inference_MS_openai.py). A final API error is no verdict.",
  "preview": true,
  "primary_metric": "bcr",
  "protocol": {
    "commit": "d7eb086c324b473c43a9de86e0e653a25d729835",
    "name": "Paired clinical error detection (Both-Correct Rate, pointwise metrics, evidence-contrast analysis)",
    "ported": "Prompts (MEDERRBENCH_EN_PROMPTS, byte for byte), messages, the tolerant and strict verdict parsers, compute_metrics, compute_trad, the pair classification, the random-assignment imputation, the evidence-contrast analysis (ECA) with TP localisation, and the expected BCR and independence ratio, in src/medbench/tasks/note_errors_protocol.py",
    "provisional": "The authors' code is used for prompts, parsing, metrics and ECA; the items (rules, clean filter, twin edits, selection) are medbench's, as no published construction exists for this dataset. Whether this transfer counts as provisional under SOP A1 is decided by the owner at publication (lead decision 3); the task is hidden until then.",
    "repository": "https://github.com/healthylaife/paired-clinical-eval",
    "source": "Zhang Y, Beheshti R. Toward Better Assessment of LLMs' Performance in Clinical Error Detection. MLHC 2026. arXiv:2608.16643 (CC BY 4.0)",
    "taxonomy": "MEDEC error categories (Ben Abacha A, et al. Findings of ACL 2025. arXiv:2412.19260), cited only"
  },
  "request": {
    "configurations": {
      "conservative-greedy": {
        "prompt": "conservative",
        "temperature": 0
      },
      "conservative-sample": {
        "prompt": "conservative",
        "temperature": 0.7,
        "top_p": 0.9
      },
      "neutral-greedy": {
        "prompt": "neutral",
        "temperature": 0
      },
      "neutral-sample": {
        "prompt": "neutral",
        "temperature": 0.7,
        "top_p": 0.9
      }
    },
    "max_tokens": 2048,
    "messages": "system + user (MEDERRBENCH_EN_PROMPTS[prompt]['user'] with the note formatted in); English instructions, Korean note",
    "overrides": {
      "zhipu/glm-5.3-753b-a40b": {
        "assistant_prefill": "<think></think>\n",
        "fields": {
          "add_generation_prompt": false,
          "chat_template_kwargs": {
            "enable_thinking": false
          },
          "continue_final_message": true
        }
      }
    },
    "temperature": 0,
    "thinking": "off, with the serving route's thinking-off fields; a per-model override replaces them after validation on train notes. Every run records a thinking audit (replies with a reasoning field, replies with a think block); thinking is off only when both are 0 (design 21.4)"
  },
  "slug": "syn-snuh-note-errors",
  "source": {
    "doi": "10.23421/2026.0770",
    "files": "synthetic_snuh.duckdb, 98,578,432 bytes, SHA-256 fd2056b31cc74c171c3906a7e48482979b993d7a52471dbe5ddd3f3df21a87c8 (medbench's; KHDP publishes no checksum)",
    "khdp_code": "SYN-SNUH",
    "note_text": "note_llm_log.text (the note Qwen3.6-35B-A3B wrote); note.note_text is a template and is not used",
    "version": "1.0.0"
  },
  "split": {
    "by": "person_id",
    "method": "sha256(salt, 'split', person_id), 50/50, with a private 32-byte salt; the 219 patients whose notes were used in the pilots, in filter development or in the GLM-5.3 format check are in train only. Neither the salt nor the split is published while a test version is live (lead decision 2).",
    "test": "620 pairs from 620 patients (the reported split)",
    "train": "620 pairs from 620 other patients; leak probes and tagged few-shot or fine-tuned runs only"
  },
  "title": "Clinical error detection in synthetic Korean SNUH notes (paired)",
  "version": 0,
  "visibility": "hidden"
}

JSON: /v1/tasks/syn-snuh-note-errors · /v1/leaderboard?task=syn-snuh-note-errors@0