Mds Icu Eval

---
name: mds-icu-eval
description: This benchmark evaluates a multimodal deep learning model's ability to predict 33 distinct ICU clinical outcomes by fusing 10-second 12-lead ECG waveforms with structured tabular clinical data. It probes the model's discriminative capacity and probabilistic calibration across mortality, medication administration, clinical deterioration, and organ dysfunction tasks. Use when the user wants to benchmark on MDS-ICU, or asks about evaluating this task. Reports macro-averaged AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.06645
  bibtex_key: lopez2026mdsicu
  confidence: high
---

# mds-icu-eval

> A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking — López Alcaraz et al. (2026) (arXiv:2601.06645, 2026)

## What this evaluates

This benchmark evaluates a multimodal deep learning model's ability to predict 33 distinct ICU clinical outcomes by fusing 10-second 12-lead ECG waveforms with structured tabular clinical data. It probes the model's discriminative capacity and probabilistic calibration across mortality, medication administration, clinical deterioration, and organ dysfunction tasks.

## Datasets

- **MDS-ICU** — total 63001; splits: train (-1), val (-1), test (-1); repo https://github.com/AI4HealthUOL/MDS-ICU

## Metrics

- `macro-averaged AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, averaged across all 33 binary prediction tasks. Computed by plotting the true positive rate against the false positive rate at various classification thresholds and calculating the area under the curve for each task, then taking the mean.

## Input / output format

**Input**: A 10-second 12-lead ECG waveform array concatenated with an 801-dimensional vector of temporal clinical features (demographics, vital signs, laboratory values, surgical/procedural context).

**Output**: A continuous probability score between 0 and 1 for each of the 33 predefined binary clinical outcomes.

## Scoring recipe

```python
def compute_macro_auroc(y_true_list, y_pred_list):
    aurocs = []
    for y_true, y_pred in zip(y_true_list, y_pred_list):
        fpr, tpr, _ = roc_curve(y_true, y_pred)
        aurocs.append(auc(fpr, tpr))
    return np.mean(aurocs)
```

## Common pitfalls

- Patient-wise stratified splitting must be strictly enforced to prevent data leakage, as multiple samples originate from the same ICU stay/patient.
- Evaluation is threshold-free (AUROC); comparing model probabilities directly to clinician binary decisions requires plotting ROC curves or using the Youden index rather than fixed accuracy cutoffs.
- Calibration (isotonic regression) is applied on the validation set before computing Brier scores, which must not be skipped when assessing probabilistic reliability.

## Evidence (verbatim from paper)

> Our approach employs S4 encoders for ECG waveforms and RealMLP for tabular features with late fusion, evaluated using stratified patient-wise splits and macro-averaged AUROC as the primary metric.

## Citation

```bibtex
@misc{lopez2026mdsicu,
  title={A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking},
  author={López Alcaraz et al. (2026)},
  year={2026},
  note={arXiv:2601.06645}
}
```

- arXiv: 2601.06645