Mimic Iii Benchmark Eval

---
name: mimic-iii-benchmark-eval
description: Evaluates clinical risk prediction and intervention forecasting using structured EHR time-series data. Probes a model's ability to handle missingness, temporal gaps, and class imbalance in ICU patient records. Use when the user wants to benchmark on MIMIC-III, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1907.08322
  bibtex_key: wang2019mimicextract
  confidence: high
---

# mimic-iii-benchmark-eval

> MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III — Wang et al. (arXiv:1907.08322, 2019)

## What this evaluates

Evaluates clinical risk prediction and intervention forecasting using structured EHR time-series data. Probes a model's ability to handle missingness, temporal gaps, and class imbalance in ICU patient records.

## Datasets

- **MIMIC-III** — total ?; splits: full (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve; computed by ranking predictions against labels and calculating the trapezoidal approximation of the ROC space. Reported per class and macro-averaged for multiclass tasks.
- `AUPRC` — range: [0, 1]
  - Area under the precision-recall curve; calculated by integrating precision over recall thresholds. Reported per class and macro-averaged.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances using a fixed 0.5 classification threshold.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall at a 0.5 threshold. Macro-averaged for multiclass intervention tasks.

## Input / output format

**Input**: First 24 hours of hourly time-series features (labs, vitals, static demographics) with a mandatory 6-hour gap before the prediction target. Features are mean-centered, scaled, and imputed using forward-fill, patient-specific mean, or global mean, accompanied by presence masks and time-since-last-observation.

**Output**: Binary label (mortality/LOS) or 4-class label (Onset, Stay On, Wean, Stay Off) for intervention prediction.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred_proba, task_type='binary', threshold=0.5):
    if task_type == 'binary':
        y_pred = (y_pred_proba >= threshold).astype(int)
        return {'AUROC': roc_auc(y_true, y_pred_proba),
                'AUPRC': average_precision(y_true, y_pred_proba),
                'Accuracy': accuracy(y_true, y_pred),
                'F1': f1(y_true, y_pred)}
    else:
        y_pred = np.argmax(y_pred_proba, axis=1)
        return {'Macro AUROC': roc_auc(y_true, y_pred_proba, average='macro'),
                'Macro F1': f1(y_true, y_pred, average='macro'),
                'Macro AUPRC': average_precision(y_true, y_pred_proba, average='macro')}
```

## Common pitfalls

- Failing to enforce the mandatory 6-hour gap between the last feature observation and the prediction target, which causes severe temporal label leakage.
- Relying solely on Accuracy for evaluation, as the authors note that class imbalance makes Accuracy misleading compared to AUPRC/F1.
- Including diagnosis/billing codes as prediction targets, which the authors explicitly exclude due to lack of temporal association with treatment.

## Evidence (verbatim from paper)

> Our AUROCs are very much in line with the literature for these tasks, showing robustly high performance for GRU-D and RF models, as expected. One interesting observation is that random forest models often have poor F1 scores, even while maintaining competitive AUPRC scores.

## Citation

```bibtex
@misc{wang2019mimicextract,
  title={MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III},
  author={Wang et al.},
  year={2019},
  note={arXiv:1907.08322}
}
```

- arXiv: 1907.08322