Biosignal › SNUH LYDUS ECG 50K › lydus-ecg-image@0

Diagnostic classes of 12-lead paper-ECG images (rendered from SNUH ECGs; PhysioNet/CinC Challenge 2024 classes) preview

Data: doi.org/10.23421/2025.0704 · CC BY 4.0 · multilabel · primary metric: macro_f_measure · spec bce2a0d6ce91

No results yet.

Data

SNUH LYDUS ECG 50K (biosignal). License: CC BY 4.0. Source: https://doi.org/10.23421/2025.0704

Seoul National University Hospital. SNUH LYDUS ECG 50K (version 1.1.1). KHDP. 2025. doi:10.23421/2025.0704

Task specification

{
  "baselines": {
    "headroom": "pending the trained baseline; the earlier ResNet-18 result would give H = (1 - 0.676) / (1 - 0.252) = 0.43",
    "medbench/lydus-ecg-image-resnet18": {
      "earlier_result": "on the previous spec hash (22c9b073c435), its test predictions scored with the current labels: macro F 0.676 (0.650-0.700), validation 0.677 (epoch 10 of 12), Acute MI F 0.190 on 23 positives",
      "status": "pending, not recorded on this spec. It was trained and thresholded on 2026-10-30 before the Acute MI rule followed the Challenge, when 56 training and 8 validation records lacked Acute MI (no other label differs). `medbench baseline lydus-ecg-image` records the CNN only when its summary.json carries the digest of the current training and validation labels (cnn_run, labels_sha256), and this network's has none. Training it again needs the training images rendered again on pods, which waits for the build's D19 listing.",
      "training": "ResNet-18 (torchvision 0.22, ImageNet weights), grey half-size images, BCE, AdamW 3e-4 cosine, batch 32, 12 epochs, AMP, seed 0 (src/medbench/baselines/lydus_ecg_image_cnn.py); one RTX 5090, about 9 minutes"
    },
    "medbench/prior": {
      "ci": [
        0.247,
        0.258
      ],
      "macro_f_measure": 0.252,
      "output": "all 11 classes"
    }
  },
  "checks": {
    "labels": "agreement with the reviewed columns and the Acute MI sources (deviations); class counts per part (counts); tests pin the Acute MI rule to the Challenge's statement list and to the shared map's lines it matches (tests/test_lydus_ecg_image.py)",
    "renderer": "the same record renders to the same bytes on one machine (seeded); images of the build were viewed",
    "scorer": "tests/test_cinc2024.py runs the vendored evaluate_model.py on WFDB folders and matches the port's numbers",
    "transfer": "every medbench-made part in protocol.medbench_made has one deviation starting with its name that says how it was checked (tests/test_lydus_ecg_image_transfer.py, with protocol_transfer.transfer_problems)"
  },
  "counts": {
    "acute_mi_sources": {
      "test": [
        16,
        7
      ],
      "train": [
        177,
        20
      ],
      "validation": [
        32,
        7
      ]
    },
    "hedged_would_add_test": {
      "HYP": 40,
      "Old MI": 91,
      "PAC": 1,
      "STTC": 26
    },
    "sttc_sources_test": [
      277,
      326,
      80
    ],
    "test": {
      "AFIB/AFL": 799,
      "Acute MI": 23,
      "BRADY": 120,
      "CD": 955,
      "HYP": 131,
      "NORM": 168,
      "Old MI": 159,
      "PAC": 58,
      "PVC": 158,
      "STTC": 683,
      "TACHY": 44,
      "groups": 1417,
      "records": 2000,
      "scored": 1780
    },
    "train": {
      "AFIB/AFL": 6588,
      "Acute MI": 197,
      "BRADY": 1010,
      "CD": 7526,
      "HYP": 1033,
      "NORM": 1068,
      "Old MI": 1365,
      "PAC": 373,
      "PVC": 1526,
      "STTC": 5761,
      "TACHY": 398,
      "groups": 9542,
      "records": 15999,
      "scored": 14259
    },
    "validation": {
      "AFIB/AFL": 822,
      "Acute MI": 39,
      "BRADY": 118,
      "CD": 965,
      "HYP": 90,
      "NORM": 156,
      "Old MI": 188,
      "PAC": 47,
      "PVC": 171,
      "STTC": 760,
      "TACHY": 51,
      "groups": 1275,
      "records": 1999,
      "scored": 1807
    }
  },
  "dataset": "lydus-snuh-ecg-50k",
  "decisions": [
    "Owner 2026-10-30: protocol transfers are formal A1 builds; LYDUS-SNUH-ECG-50K is SNUH patient data; sending SNUH-derived ECG images to external model APIs is allowed (CC BY 4.0); this task's check may spend up to $20 on paid image runs, enforced in code (vision.IMAGE_CHECK_CAP_USD; run_image_task refuses a larger cap). Both are SOP rules for image tasks (docs/sop/benchmarks-and-models.md)."
  ],
  "deviations": [
    "Data: SNUH ECGs (LYDUS-SNUH-ECG-50K 1.1.1) instead of PTB-XL.",
    "Label map: the release's dx lines (the electrocardiograph's statements, GE 12SL for most records) become SNOMED CT codes of the Challenge 2021 tables through the shared LYDUS map that medbench made for lydus-ecg12-dx@0 (registry/maps/lydus-dx-snomed-v0.csv), definite codes only; the Challenge read PTB-XL's cardiologist statements. Hedged statements (\"Possible\", \"Cannot rule out\", \"consider\", \"suspected\") are not labels, while PTB-XL's mapping kept statements of every likelihood: including them would add 91 Old MI, 40 HYP, 26 STTC and 1 PAC labels in test. Checked: against the release's columns reviewed by two clinicians, over all 50,000 records. Agreement is the share of records in which the class is present exactly when the column names it: AFIB/AFL 0.967 (rhythm A.fib or A.flutter), BRADY 0.997 (S.brady), TACHY 0.999 (S.tachy or SVT), PVC 0.968 (premature beat VPC), PAC 0.977 (APC), CD 0.987 (any entry of the bundle branch block column; 1AVB, 2AVB, 3AVB or IVB in the AV block column; or rhythm WPW).",
    "Classes: the Challenge derives NORM, MI, STTC, CD and HYP from PTB-XL's cardiologist statements grouped into its diagnostic superclasses and the rhythm and beat classes from PTB-XL statements (prepare_ptbxl_data.py); here they come from the label map's SNOMED CT codes, grouped in SNOMED_CLASS on the model of PTB-XL's superclasses but not the same: STTC also takes the ST and T form findings PTB-XL keeps out of it (TAb, STIAb, TInv, STD, STE; 12SL words most nonspecific ST-T changes that way), and the 12SL injury statements (injury patterns, subendocardial injury) give STTC but no MI class, while PTB-XL puts its INJ* statements in its MI superclass, which the Challenge labels Old MI unless an acute statement is present. Checked: each code group was compared with the PTB-XL statements of its superclass in scp_statements.csv (the code names them), and a test checks that every mapped code has a Challenge 2024 class. The differences were counted on the test set: 406 of the 683 STTC positives have none of PTB-XL's STTC findings (326 with T wave abnormal, 80 with only STIAb, TInv, STD or STE), and 26 records carry injury statements, 22 of them with no MI class.",
    "Acute MI: the Challenge's 12SL statement rule is applied to the dx text word for word, with one addition: the older wording \"*** ** ** ** * ACUTE MI ** ** ** **\", which PTB-XL+'s 12SL table does not hold (so its statement number, and whether the Challenge's list covers it, is unknown), also counts; it states acute MI outright, and without it 7 test, 20 training and 7 validation records would be Old MI through its acute-MI code. The line \"ST elevation, consider early repolarization\" matches none of the listed texts and does not count (4 test and 11 training records). \"probably acute MI (anterior / inferior / lateral ...)\" is hedged in the shared map and gives no Acute MI (15 records of the release; 1 test, 7 training and 1 validation record in the build). As in the Challenge, Acute MI does not need an infarct statement, and a hedge does not remove it. Checked: tests pin the statement list to the Challenge's list and PTB-XL+'s table, and to the 8 lines of the shared map it matches; the labels of all 19,998 instances are reproduced from the release. Acute MI positives: test 23, training 197, validation 39 (16, 177 and 32 from the Challenge's statements; 7, 20 and 7 from the older wording alone).",
    "Exclusions: records the electrocardiograph did not interpret (2: training 1, validation 1) and one whose 8 stored leads are all flat (training) are left out, as in lydus-ecg12-dx@0. Checked: the build counts each rule's records per part (build-summary.json, left_out); no test record is left out.",
    "Split: the Challenge trained on PTB-XL and scored hidden validation and test sets; here every part comes from the shared LYDUS split (medbench-made in lydus-ecg12-dx@0, sha256 above), by groups of likely repeat ECGs, because the release has no patient id. Checked: `select` refuses a split file whose sha256 differs from the spec's; no group spans test, training and validation (0 of the build's 12,234 groups).",
    "Records: 2,000 test records sampled from the shared LYDUS test split (cost per image run), not a full split, and 16,000 training and 2,000 validation records for the baselines. Checked: a test draws the sample from a toy split (the same records for the same seed whatever the row order; the test sample does not move when a training size changes); the build has 2,000 test records in 1,417 groups, 1,780 of them scored.",
    "Leads: the release stores I, II and V1-V6, so III, aVR, aVL and aVF are derived before rendering (III = II - I, aVR = -(I + II) / 2, aVL = I - II / 2, aVF = II - I / 2), as in lydus-ecg12-dx@0; PTB-XL stores all 12. Checked: a test derives the four leads from known I and II and compares them in mV.",
    "Images: one pinned generator setting (input.render), without handwriting, wrinkles or rotation; the Challenge's images used random settings and its hidden set added real scans. Rendering is seeded per record; the stored image is a JPEG. Checked: a test compares the spec's settings and kit commit with those the code passes (lydus_ecg_render.RENDER); a record rendered twice on one machine gave the same bytes; the build's images were viewed; render parameters do not differ by class (test images with AFIB/AFL: mean noise level 25.96 against 25.69 for the others, mean file size 1,633 against 1,617 KB).",
    "Intervals: a bootstrap over groups of likely repeat ECGs, 1,000 resamples each keeping the full reference's classes (the Challenge reports no interval). Checked: tests/test_cinc2024.py (whole groups are resampled; the interval brackets the point value, which equals the vendored evaluate_model.py's).",
    "Prompt and parser: language models get PROMPT (id lydus-ecg-image-v0: the 11 classes with one-line definitions) and answer in JSON, parsed by PARSER_VERSION labels-json-v0 (the Challenge's entrants were trained models that wrote label files); a model with no answer for an image is scored as predicting no class, as the Challenge scores a missing output. Checked: tests (the prompt names every class; parser cases; an unparsed answer scores as no labels); the free pilot's 100 answers parsed at 0.92 (the other 8 used all 16,384 tokens) with no route error."
  ],
  "exclusions": {
    "flat_signal": "all 8 stored leads constant: 1 selected record (train)",
    "no_analysis": "the electrocardiograph made no interpretation: 2 selected records (train 1, validation 1)"
  },
  "input": {
    "image": "one JPEG per record (quality 90, 2200 x 1700 pixels): the 10-second ECG printed by ECG-Image-Kit in the standard 3 x 4 layout (2.5 s per lead: I, II, III | aVR, aVL, aVF | V1, V2, V3 | V4, V5, V6) with a 10-second lead II rhythm strip, 25 mm/s, 10 mm/mV, lead names and calibration pulses, at 200 dpi",
    "leads": "the release stores I, II and V1-V6 (int16, 500 Hz, about 4.88 uV per count); III, aVR, aVL and aVF are derived (III = II - I, aVR = -(I + II) / 2, aVL = I - II / 2, aVF = II - I / 2) before rendering, as in lydus-ecg12-dx@0",
    "never": "new_id or any record id (instances use opaque ids), the dx text, the reviewed rhythm, premature beat, bundle branch block and AV block columns, age, sex and the electrocardiograph's measurements: none is printed on the image (no header text)",
    "render": {
      "per_record": "seed = sha256(\"20261030:\" + instance_id) as a 31-bit integer; Python's random, NumPy's global generator and imgaug are seeded with it before each record; the font comes from the sorted font list; the drawn noise level, colour temperature and font of every test image are kept in render-params.jsonl",
      "script": "src/medbench/tasks/lydus_ecg_render.py",
      "settings": {
        "augment": true,
        "black_and_white": 0.0,
        "calibration_pulse": 1.0,
        "columns": -1,
        "crop": 0.01,
        "full_mode": "II",
        "grid_present": 1.0,
        "handwritten_text": false,
        "lead_names": true,
        "mask_unplotted_samples": true,
        "noise_max": 50,
        "pad_inches": 0,
        "print_header": false,
        "resolution_dpi": 200,
        "rotate_deg": 0,
        "standard_grid_color": 5,
        "wrinkles": false
      },
      "stored_as": "JPEG quality 90 (the generator's PNG with its added noise is about 8 MB)",
      "wfdb_gain": "each record goes to the renderer as WFDB format 16 at 1 uV per count; the 6 selected records with a value beyond +-32.767 mV (5 train, 1 validation; no test record) at 10 uV (lydus_ecg_render.record_gain)"
    }
  },
  "kind": "multilabel",
  "label": {
    "classes": {
      "AFIB/AFL": "atrial fibrillation or flutter (164889003, 164890007, 282825002)",
      "Acute MI": "the Challenge's rule, applied to the dx text: any of its acute-MI 12SL statements, 821 \"** ** ACUTE MI / STEMI ** **\", 902 \"ST elevation, consider early repolarization, pericarditis, or injury\", 903 \"ST elevation, probably due to early repolarization\" or 963-967 \"ST elevation, consider <inferior, anterior, lateral, anterolateral, inferolateral> injury or acute infarct\" (prepare_ptbxl_data.py lists 821, 822, 823, 827, 829, 902, 903, 904 and 963-968 and keeps those PTB-XL+'s 12SL table holds, which are these; texts compared ignoring case, spacing and punctuation other than * and /), whatever the record's other statements; and the older wording \"*** ** ** ** * ACUTE MI ** ** ** **\" (a deviation, below)",
      "BRADY": "sinus bradycardia (426177001)",
      "CD": "conduction disturbance (bundle branch and fascicular blocks, 1st-3rd degree AV block, nonspecific intraventricular conduction disorder, WPW; 16 codes)",
      "HYP": "hypertrophy (left and right ventricular hypertrophy, left and right atrial enlargement or hypertrophy)",
      "NORM": "normal ECG (the 12SL statement \"Normal ECG\" or \"Otherwise normal ECG\")",
      "Old MI": "a myocardial infarction code (164865005, 54329005, 164867002, 57054005: PTB-XL's MI superclass) without an acute-MI statement",
      "PAC": "premature atrial or supraventricular complexes (284470004, 63593006)",
      "PVC": "premature ventricular complexes (427172004, 17338001, 251182009, 11157007)",
      "STTC": "ST/T change (164934002, 164930006, 428750005, 111975006, 59931005, 429622005, 164931005, 164861001, 425623009)",
      "TACHY": "sinus or supraventricular tachycardia (427084000, 426761007, 67198005)"
    },
    "map_code": "src/medbench/tasks/lydus_ecg_image.py (SNOMED_CLASS, MI_CODES, ACUTE_MI_STATEMENTS, ACUTE_MI_OLDER_WORDING, NORM_STATEMENTS)",
    "not_a_class": "as in PTB-XL's superclasses, form statements (left ventricular high voltage, Q waves, low QRS voltage, axis deviation), pacing, junctional and other rhythms map to no class; a record with no class is not scored (the Challenge skips records without labels) but stays in the test set",
    "source": "the shared LYDUS label map of lydus-ecg12-dx@0 (registry/maps/lydus-dx-snomed-v0.csv): each record's dx lines (the electrocardiograph's statements) to SNOMED CT codes of the Challenge 2021 tables; definite codes only (a statement qualified \"Possible\", \"Cannot rule out\", \"consider\", \"suspected\" is hedged and is not a label)"
  },
  "leakage_checks": {
    "ids": "instance ids are sha256(\"lydus-ecg-image@0:\" + new_id)[:12]; images carry no header text",
    "render": "render parameters and file sizes do not differ by class (Images deviation)",
    "split": "the test sample comes from the shared split's test groups only; no group spans parts"
  },
  "limitations": [
    "Synthetic images are easier than photographs or scans of paper ECGs: no handwriting, creases, wrinkles, rotation, shadows or perspective, one layout and one paper colour scheme; colour temperature, additive noise and a crop of at most 1% vary. The Challenge's hidden set also held real scans and photographs.",
    "Labels are the electrocardiograph's statements (GE 12SL for most records), reviewed by two clinicians in the release, not a cardiologist's over-read. The reviewed rhythm and beat columns agree with the mapped classes on 96.7-99.9% of the 50,000 records (the label map deviation gives each class and how agreement is defined); PAC and PVC are named in many more statements than in the reviewed premature-beat column (1,216 and 4,624 records against 101 and 3,076).",
    "The release enriches abnormal rhythms and conduction (AFIB/AFL 40%, CD 48% of test records); NORM is 8%. Acute MI has 23 test positives, so its F-measure, which counts as much as any class, is noisy."
  ],
  "metrics": {
    "bootstrap": {
      "by": "group",
      "n": 1000
    },
    "by_class": [
      "f_measure",
      "n_positive"
    ],
    "primary": "macro_f_measure",
    "secondary": [
      "parse_rate",
      "error_rate",
      "mean_labels_predicted"
    ]
  },
  "pilot": {
    "alibaba/qwen3.8-flash-next-180b-a6b": "first 100 test images by instance id (a check run, not listed). Its 100 replies of 2026-10-30 (the run after the Acute MI fix) were scored again on this spec, since the prompt, images, labels and parser are unchanged: macro F 0.259 (0.175-0.326; prior on the same images 0.260), parse rate 0.92 (8 answers reached 16,384 tokens without a label object), no route errors, AFIB/AFL F 0.105 on 36 positives; thinking left at the route's default; free route, no paid run"
  },
  "prediction": "for each image, zero or more of the 11 class names. Language models answer with JSON {\"labels\": [...]}; the last such object in the answer counts, class names case-insensitive, other names ignored; an answer without one counts as no labels (reported as the parse rate)",
  "preview": true,
  "primary_metric": "macro_f_measure",
  "prompt": "src/medbench/tasks/lydus_ecg_image.py (PROMPT, id lydus-ecg-image-v0; temperature 0, at most 16,384 tokens)",
  "protocol": {
    "adopted": "The classification task of the George B. Moody PhysioNet Challenge 2024 (digitisation and classification of ECG images): from the image of a printed 12-lead ECG, name every class that applies out of 11, scored by the Challenge's macro F-measure. The Challenge used images of PTB-XL ECGs made with ECG-Image-Kit; this task applies the Challenge's scoring code, at a pinned commit, to images of SNUH ECGs (LYDUS-SNUH-ECG-50K) made with the same generator at a pinned commit and settings. It is a protocol transfer (owner decision 2026-10-30: a formal build, whose scores may be published, because the scoring code is the Challenge's own and every medbench-made part is listed under deviations and checked).",
    "commit": "9af7dfbc3a8af209638d982b344cce28c67abe4e",
    "commit_note": "9af7dfb (2024-09-13) is the scorer of the Challenge's final results. The repository's head, 1a51354 (2025-10-17, \"Clarify shift\"), is not used: for a negative time shift it keeps only the first |shift| samples of a digitised lead (x_est[:-offset_hz]), scoring a slightly early digitisation near 0 dB; the classification code is the same.",
    "dataset_release": "LYDUS-SNUH-ECG-50K 1.1.1, KHDP, doi:10.23421/2025.0704, CC BY 4.0 (ECGs of Seoul National University Hospital patients, 2010-2020; the owner confirmed SNUH patient data on 2026-10-30)",
    "label_rules": "https://github.com/physionetchallenges/python-example-2024 @ e19f14bb18d4ce86f4137f07cfd44243c0692e8e, prepare_ptbxl_data.py (how PTB-XL statements become the 11 classes; Acute MI from its list of 12SL statement numbers, read in PTB-XL+ 1.0.1 labels/mapping/12slv23ToSNOMED.csv)",
    "medbench_made": [
      "label map",
      "classes",
      "Acute MI",
      "exclusions",
      "split",
      "records",
      "leads",
      "images",
      "intervals",
      "prompt and parser"
    ],
    "renderer": {
      "commit": "27b90f56896c9fc78b05a83ca14844ea2637aa0b",
      "paper": "Shivashankara KK, Deepanshi, Shervedani AM, Reyna MA, Clifford GD, Sameni R. ECG-Image-Kit: a synthetic image generation toolbox to facilitate deep learning-based electrocardiogram digitization. Physiol Meas. 2024. doi:10.1088/1361-6579/ad4954",
      "repository": "https://github.com/alphanumericslab/ecg-image-kit"
    },
    "repository": "https://github.com/physionetchallenges/evaluation-2024",
    "source": "Reyna MA, et al. Digitization and classification of ECG images: the George B. Moody PhysioNet Challenge 2024. Computing in Cardiology 2024. Challenge page: https://physionetchallenges.org/2024/ (classes and scoring)",
    "transfer": true
  },
  "slug": "lydus-ecg-image",
  "split": {
    "by": "group",
    "code": "src/medbench/tasks/lydus_ecg_image_cli.py (select_records)",
    "sample": "a seeded random sample of each part of the shared split (numpy default_rng([20261030, k]) over the part's records sorted by new_id; k = 0 test, 1 train, 2 validation): test 2,000 of 10,000 records (image runs cost money per item), train 16,000 of 35,000 and validation 2,000 of 5,000 for the baselines",
    "sha256": "a7fddba5764a73d8c1c936296fa52909a822b797be321ec9448c7cc345ea3779",
    "shared": "registry/splits/lydus-snuh-ecg-50k-1.1.1-split.csv"
  },
  "title": "Diagnostic classes of 12-lead paper-ECG images (rendered from SNUH ECGs; PhysioNet/CinC Challenge 2024 classes)",
  "version": 0,
  "visibility": "hidden"
}

JSON: /v1/tasks/lydus-ecg-image · /v1/leaderboard?task=lydus-ecg-image@0