Structured EHR › Synthetic K-MIMIC (SYN-ICU) › syn-icu-sofa@0

First-day SOFA from a Korean ICU record (SYN-ICU) preview

Data: khdp.net/database/data-search-detail/723/SYN-ICU/2.0.0 · CC BY 4.0 · score · primary metric: exact_match · spec 0c530cb69b70

No results yet.

Data

Synthetic K-MIMIC (SYN-ICU) (structured). License: CC BY 4.0. Source: https://khdp.net/database/data-search-detail/723/SYN-ICU/2.0.0

Seoul National University Hospital. Synthetic K-MIMIC (SYN-ICU), version 2.0.0. KHDP. 2025. doi:10.23421/2025.0723

Task specification

{
  "baselines": {
    "medbench/prior": "each field's most frequent value among KMM train patients' included stays (SOFA total 4)",
    "medbench/syn-icu-sofa-hgb": "HistGradientBoostingRegressor per field on summary features of the same frames (per item minimum, maximum, mean, count; intake and output sums; ventilator mode counts; drug counts and maximum rates; age, sex, weight), rounded to the nearest integer within the training range, without any SOFA threshold; 5-fold cross-fitting over patients (seed 0), so every stay is predicted by a model that never saw it"
  },
  "data": {
    "source": "KHDP SYN-ICU 2.0.0 (doi:10.23421/2025.0723, CC BY 4.0), 15 xlsx tables, SHA-256 pinned in syn_icu.FILES",
    "split_source": "KHDP KMM 1.0.0 (K-MIMIC-MEDS, doi:10.23421/2026.0762, CC BY 4.0): subject_splits.parquet",
    "what": "GAN-generated (EHR-safe) ICU stays from SNUH's ICU registry K-MIMIC: 1,328 stays, one per patient, each with up to 30 hourly frames from ICU admission (hour 0 to 29). No table holds data before ICU admission, so mimic-code's 6-hour look-back before admission finds nothing."
  },
  "dataset": "syn-icu",
  "deviations": [
    "Item map: medbench.kr maps 17 SYN-ICU chart items, 8 lab items, 2 urine output items and the 9 ventilator mode values to the MIMIC-IV item ids and values the SQL reads (syn_icu_sofa_labels.CHART_ITEMS, LAB_ITEMS, OUTPUT_ITEMS, VENT_MODES). All seven SYN-ICU modes other than STANDBY and Others map to MIMIC modes that ventilation.sql counts as invasive ventilation; STANDBY and Others map to modes it leaves uncategorised. The combined urine and stool volume (소변+대변), CVVHDF fluid removal and drains are not urine. Checked: each item against the release's label and unit (d_items, d_labitems) and its value distribution against the MIMIC item's unit (medians: heart rate 84/min, platelets 147 x10^3/uL, creatinine 1.14 mg/dL, total bilirubin 1.38 mg/dL, PaO2 122 mmHg, FiO2 43%, Foley urine 95 mL per hour).",
    "Specimens: SYN-ICU numbers every lab row as its own specimen, so medbench.kr groups a stay's lab rows drawn at the same time into one specimen, and marks a specimen with a PaO2 as arterial ('ART.'), since SYN-ICU records no specimen type and bg.sql scores arterial gases only. Without the grouping no PaCO2 would ever share a specimen with a PaO2 and the SIRS PaCO2 rule could never fire. Checked: pO2, pCO2 and pH of one draw share an hour (in frames 0-24 h of all 1,328 stays, 2,762 hours have two or all three of them).",
    "Vasopressors: SYN-ICU charts norepinephrine and epinephrine (emar_detail.infusion_rate_min) with no unit, so their rows enter the SQL with no rate and first_day_sofa.sql is patched to score their presence in the window as 3, the score the code gives any rate of 0.1 ug/kg/min or less; the 4 for more than 0.1 cannot be given. Dopamine and dobutamine are not in SYN-ICU. The unit could not be established, so this fallback stays: no documentation names it (KHDP metadata of SYN-ICU 1.0.0 and 2.0.0, KMM 1.0.0, K-MIMIC 1.0.0 and K-MIMIC-MORTAL 1.0.0, whose data dictionary link is empty; KMM's codes, and its MEDICATION events carry no value; the K-MIMIC-MEDS ETL at 61fb383 passes emar_detail through without using its rates), and the column holds each drug's own dosing unit rather than one unit (all 5,675 emar rows): propofol 25-36 and remifentanil 0.01-0.06 read as ug/kg/min, vasopressin 0.03-0.04 as units/min, dexmedetomidine 0.35-0.40 as ug/kg/h (per minute it would be 21 ug/kg/h, 15 times the usual maximum), while fluids and insulin sit in infusion_rate_hr. Norepinephrine (median 6.67, 0.61-67.2, 1,045 rows) cannot be ug/kg/min and looks like ug/min, a per-minute amount that rises with weight (Spearman 0.24 over 498 stays, against -0.16 for propofol and 0.04 for remifentanil); epinephrine (median 0.39, 0.27-0.45, 23 rows) does not, since as ug/min it would be about 0.006 ug/kg/min. Reported both ways: 462 of the 1,293 included stays received one of the two in frames 0-24 h; with norepinephrine read as ug/min divided by the stay's weight, 333 of them would score 4 (none from epinephrine read the same way; 341 if epinephrine were ug/kg/min). Checked: the patch is four exact text replacements, each required to match once (syn_icu_sofa_labels.PATCHES).",
    "Window: only the frames 0-24 h after ICU admission reach the SQL (the frames the model sees). On SYN-ICU this is what the first_day windows read anyway, except where a ventilation episode runs past 24 h: with all 30 frames, 7 of 1,328 stays would get another respiration score. Checked by computing the labels both ways.",
    "qSOFA port: mimic-iv has no qSOFA; mimic-iii's qsofa.sql (same commit) is run with its inputs renamed to the mimic-iv first-day tables (sysbp_min -> sbp_min, resprate_max -> resp_rate_max, mingcs -> gcs_min, icustay_id -> stay_id). mimic-iv's sbp averages non-invasive and arterial systolic pressure taken at the same time. Checked: six exact text replacements, each required to match once.",
    "Cohort: 35 of 1,328 stays are left out for a physically impossible value in an item the scores read (GAN data), judged as the vendored SQL reads it (syn_icu_sofa.implausible): systolic below diastolic pressure (11 stays), a mean pressure outside its systolic-diastolic pair (26), or a PaO2 above the alveolar oxygen pressure at the FiO2 bg.sql pairs with it (9); 11 stays have two reasons, and the reason is in the stay list. A FiO2 that bg.sql reads as 20% up to below 21% is read as room air, 21%, before the model or the SQL sees it (syn_icu_sofa_labels.room_air; lead ruling, 2026-10-31): two stays chart one (20.0 at 19 h in stay 322958, 20.1 at 9 h in stay 412214), it was their only reason for exclusion, and both are included with unchanged labels. Values the SQL drops never exclude a stay and rescaled ones are judged rescaled: FiO2 as bg.sql reads it (above 0.2 to 1 as a fraction, 20-100 as a percentage, others dropped), pressures outside vitalsign.sql's ranges dropped (a systolic pressure of 400 or more, a diastolic or mean pressure of 300 or more), and platelet counts of 0 or less, which complete_blood_count.sql drops, are no reason (8 stays chart one, 7 of them included); the rule sheet states the same filters. Other GAN oddities are kept and documented (limitations). Checked: each rule against the SQL filter it mirrors, the pressure bounds at and below each bound (tests), and on all stays; 1,293 stays remain.",
    "Split: all included stays are test items (zero-shot task, nothing is trained on SYN-ICU); KMM's patient split is carried per stay for models trained on KMM. Checked: every SYN-ICU subject maps to exactly one KMM subject id (SHA-256 of the id, 1,328 of 1,328), one stay per patient, so no patient is in two KMM parts.",
    "Prompt: both prompts are medbench.kr's, in Korean. The headline prompt follows MedCalc-Bench's zero-shot prompt (the calculation named, reasoning allowed, a JSON answer at the end) with the stay's tables in place of a patient note, nine fields instead of one, and a conventions note that states where the labels depart from a textbook reading (request). Read the textbook way instead, one departure at a time, these change a field on the 1,293 included stays: GCS only at hours with all three components charted, CNS on 657 stays (each component carried forward without a time limit, 364); a ventilated PaO2/FiO2 of 200 or more scored by the 400 and 300 bands, respiration on 267; the latest FiO2 at any earlier hour, 142, or room air when none in 4 h, 18; ventilated whenever an invasive mode is charted at that hour, 113, or at any hour of the day, 338; a qSOFA mentation point for any GCS below 15, qSOFA on 224; the lowest single systolic pressure instead of the hourly average, qSOFA on 62, and the same for mean pressure, cardiovascular on 12; SIRS's PaCO2 from any pCO2, 30; vasopressin counted as a vasopressor, 7; urine over 24 rows, 5; lab values of 0 or less used, coagulation on 3. Two finer readings, which the prompts now rule out in words: a FiO2 charted exactly 4 h before the gas left out, respiration on 32; a GCS component carried on from the previous GCS hour where it was itself filled, CNS on 7 and qSOFA on 11 (18 stays). The rule sheet (report track) is medbench.kr's statement of everything the SQL computes (windows, value filters, GCS carry-forward, ventilation episodes, PaO2/FiO2 pairing, thresholds and the vasopressor rule). Checked: the rule sheet as code, reading only the prompt's tables, reproduces all nine fields of all 1,293 included stays, and so does the headline prompt's reading as code, the standard definitions with the note (`medbench build syn-icu-sofa`); no label depends on data the model is not shown or on a rule the headline prompt leaves out.",
    "Answer format: one JSON object with nine integers (MedCalc-Bench asks for one answer per item); the headline is the SOFA total alone. Checked: the parser on the pilot replies.",
    "The hourly sofa.sql of mimic-code is not used: it scores a rolling 24-hour SOFA on an hourly grid from the first heart-rate chart with weight-normalised urine rates, a different quantity from first-day SOFA."
  ],
  "instances": {
    "input": "the stay's frames from ICU admission (0 h) to 24 h, both included, as a Korean-labelled text table in the release's own item labels and units: one row per hour with every charted item, intake and output volumes and drug infusion rates (as SYN-ICU records them), then the lab results (hour, test, value, unit); age, sex, weight and the ICU. A FiO2 of 20.0 or 20.1 is shown as 21 (room air; see the cohort deviation). Not shown: procedureevents (SYN-ICU charts all 11 procedures every hour for every stay), diagnoses, admission and discharge data.",
    "label": "sofa (total), respiration, coagulation, liver, cardiovascular, cns, renal, sirs, qsofa from the vendored SQL on the same frames; a component the SQL leaves NULL (no data) is 0, as the SQL's total counts it.",
    "unit": "one ICU stay (instance_id stay-<stay_id>)"
  },
  "kind": "score",
  "license": "CC BY 4.0 (SYN-ICU, KMM); mimic-code SQL MIT",
  "limitations": [
    "The data are GAN-generated: variables are often inconsistent with each other. Of the 462 included stays that received norepinephrine or epinephrine in frames 0-24 h, 348 have an mbp_min of 75 mmHg or more (first_day_sofa's minimum mean pressure, from vitalsign.sql over the same frames); 65 included stays chart GCS motor but never verbal, and 68 chart an invasive ventilator mode but no FiO2. Scores describe the rule applied to these records, not the patients' severity.",
    "The labels follow mimic-code's definition, which differs from the textbook one in places (prompt deviation). The headline prompt states those departures in a short note but not the textbook thresholds, so it measures applying SOFA, SIRS and qSOFA to a raw 24-hour record under stated conventions; a model that ignores the note and reads the record the textbook way loses points on respiration and CNS above all. The rule-sheet report track, which states every rule, measures carrying out a fully specified calculation.",
    "The cardiovascular component cannot reach 4 (no vasopressor doses): a stay on a high norepinephrine dose scores 3.",
    "SYN-ICU is one synthetic release from one hospital's ICU registry; 1,293 stays."
  ],
  "metrics": {
    "ceiling": "1.0. The rule sheet run as code on the prompt's tables alone (src/medbench/tasks/syn_icu_sofa_rules.py) reproduces every field of every included stay, so every label depends only on what the model is shown. The headline prompt's reading run as code (the standard definitions with its conventions note, without the value ranges only the rule sheet states: scores_from_prompt with value_filters=False) also reproduces every field of every included stay, so the conventions note leaves out nothing a label depends on (`medbench build syn-icu-sofa`, note_disagreements). A model with a code tool can reach the ceiling; tool use is a separate track, not this task.",
    "exact_match": "the SOFA total is correct when round(answer) equals the label (MedCalc-Bench's rule for integer-valued scores); a reply without a parseable answer is wrong",
    "intervals": "95%, bootstrap over patients (one stay per patient), 1,000 resamples, numpy seed 0",
    "primary": "exact_match",
    "secondary": "accuracy of each SOFA component, of SIRS and of qSOFA (same rule); all six components correct; mean absolute error of the total over parsed replies; total within 1; parse rate; exact match on KMM held_out patients; the rule-sheet report track's exact match on the same stays (reported, never ranked)"
  },
  "parser": "drop <think> blocks, then the last balanced JSON object with a sofa field",
  "preview": true,
  "primary_metric": "exact_match",
  "protocol": {
    "adopted": "The first-day SOFA, SIRS and qSOFA of MIT-LCP/mimic-code, its own DuckDB SQL at the pinned commit, vendored unchanged under third_party/mimic-code/ and run with the two listed patches (first_day_sofa.sql for vasopressors without a rate, mimic-iii's qsofa.sql renamed to the mimic-iv first-day tables), on SYN-ICU tables reshaped into the MIMIC-IV tables it reads; MedCalc-Bench's exact-match rule for integer-valued calculators (round the answer, compare; protocol.scorer) for the model's answer. This is a protocol transfer: neither group ever used SYN-ICU, and every part medbench.kr made for this dataset is listed under deviations.",
    "commit": "303d26c623dcc9c49cc0f204468d4acc2f063797",
    "files": [
      "mimic-iv/concepts_duckdb/firstday/first_day_sofa.sql (patched for vasopressors without a rate, see deviations)",
      "mimic-iv/concepts_duckdb/score/sirs.sql",
      "mimic-iii/concepts_duckdb/severityscores/qsofa.sql (ported to the mimic-iv first-day tables)",
      "their inputs: measurement/{bg, blood_differential, chemistry, coagulation, complete_blood_count, enzyme, gcs, oxygen_delivery, urine_output, ventilator_setting, vitalsign}, medication/{dobutamine, dopamine, epinephrine, norepinephrine}, treatment/ventilation, firstday/{first_day_bg_art, first_day_gcs, first_day_lab, first_day_urine_output, first_day_vitalsign}"
    ],
    "license": "mimic-code: MIT, Copyright (c) 2019 MIT Laboratory for Computational Physiology; the SQL files used are vendored with the licence in third_party/mimic-code/. MedCalc-Bench has no licence: its exact-match rule is reimplemented from the paper and evaluation/evaluate.py at protocol.scorer.commit (src/medbench/exact_match.py), no code is copied.",
    "medbench_made": [
      "item map",
      "specimens",
      "vasopressors",
      "window",
      "qSOFA port",
      "cohort",
      "split",
      "prompt",
      "answer format"
    ],
    "name": "mimic-code first-day SOFA, SIRS and qSOFA (MIT-LCP), scored with the MedCalc-Bench exact-match rule",
    "repository": "https://github.com/MIT-LCP/mimic-code",
    "scorer": {
      "commit": "db5649de4aec73243eaef5261e36c1b47d8707d5",
      "file": "evaluation/evaluate.py",
      "name": "MedCalc-Bench exact match for integer-valued calculators (evaluate.py, Output Type: integer A)",
      "note": "The last commit that changed evaluation/evaluate.py; the repository head on 2026-10-30 (20b10f9d66a8c3b0356ae3138533f5ec91666b6a) holds the same file (blob 41d17934). The repository has no licence, so nothing is copied.",
      "repository": "https://github.com/ncbi-nlp/MedCalc-Bench",
      "rule": "round(answer) == ground truth; an answer that is not a number is wrong"
    },
    "source": "Johnson AEW, Stone DJ, Celi LA, Pollard TJ. The MIMIC Code Repository: enabling reproducibility in critical care research. J Am Med Inform Assoc. 2018;25(1):32-39. doi:10.1093/jamia/ocx084. Khandekar N, Jin Q, Xiong G, et al. MedCalc-Bench: Evaluating Large Language Models for Medical Calculations. NeurIPS 2024 Datasets and Benchmarks. arXiv:2406.12036",
    "transfer": true
  },
  "request": {
    "max_tokens": "32,768, or the model's registry limit when it has one (thinking models need room: one probe reply used 15,226 tokens)",
    "messages": "system: the brief prompt (syn-icu-sofa-ko-brief-2 in src/medbench/tasks/syn_icu_sofa.py; Korean, in MedCalc-Bench's zero-shot style), which asks for the scores by their standard definitions, a component without data 0, followed by a conventions note that states only where the label code departs from a textbook reading: every item from the worst value over the 25 rows 0-24 h, cuff and arterial pressures charted at the same hour averaged, lab values of 0 or less unused; respiration as first_day_sofa.sql scores it (PaO2/FiO2 with the latest FiO2 charted at or up to 4 h before the gas (a FiO2 exactly 4 h earlier pairs), with a worked example, and no room-air default; a ventilated ratio scores 4 below 100 and 3 below 200 and nothing at 200 or more; a ratio off the ventilator scores 2 below 300, also below 200, and 1 below 400; ventilated means inside an episode of the seven invasive modes charted less than 14 h apart, a single record no episode); norepinephrine or epinephrine scores cardiovascular 3 because there is no ug/kg/min dose, other drugs such as vasopressin do not count; GCS totalled at every hour with a component charted, a missing component taken from the previous GCS hour alone (the nearest earlier hour with any component, if less than 6 h earlier) only where that component was charted there, never a value that was itself filled, otherwise normal (E4, V5, M6), recorded values used under sedation; urine is spontaneous voiding plus the indwelling catheter; SIRS's PaCO2 only from a gas with a PaO2; qSOFA's mentation point at a GCS of 13 or less. Textbook thresholds are not restated. user: the stay's tables. Report track, never on the leaderboard: the same stays with the rule sheet (syn-icu-sofa-ko-1) as the system prompt, which states every rule the SQL applies. Retired: the brief prompt without the note (syn-icu-sofa-ko-brief-0; lead ruling, 2026-10-31, revised the same day), and syn-icu-sofa-ko-brief-1 and syn-icu-sofa-ko-0, whose FiO2 window (\"within 4 h\") and GCS filling (\"the previous GCS hour's value\") were looser than the SQL (review of PR #110)",
    "temperature": 0
  },
  "slug": "syn-icu-sofa",
  "split": {
    "file": "registry/splits/syn-icu-sofa-0-stays.csv",
    "rule": "test-only for models that are not trained on SYN-ICU: all 1,293 included stays are test items. Each stay carries KMM's patient split (train 1,032, tuning 130, held_out 131 included stays); exact match is also reported on the held_out patients, and the trained baseline is also fitted on KMM train and tuning and scored on held_out."
  },
  "title": "First-day SOFA from a Korean ICU record (SYN-ICU)",
  "version": 0,
  "visibility": "hidden"
}

JSON: /v1/tasks/syn-icu-sofa · /v1/leaderboard?task=syn-icu-sofa@0