Mimic Iv Clinical Prediction Eval

---
name: mimic-iv-clinical-prediction-eval
description: Evaluates clinical prediction models on irregular, sparse time-series data from the ICU module of MIMIC-IV. It probes the ability of models to handle temporal sparsity, missingness, and heterogeneous features for binary classification tasks like in-ICU mortality and length of stay prediction. Use when the user wants to benchmark on MIMIC-IV 2.2, or asks about evaluating this task. Reports AUC-ROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.15290
  bibtex_key: bui2024benchmarking
  confidence: high
---

# mimic-iv-clinical-prediction-eval

> Benchmarking with MIMIC-IV, an irregular, spare clinical time series dataset — Bui et al. (2024) (arXiv:2401.15290, 2024)

## What this evaluates

Evaluates clinical prediction models on irregular, sparse time-series data from the ICU module of MIMIC-IV. It probes the ability of models to handle temporal sparsity, missingness, and heterogeneous features for binary classification tasks like in-ICU mortality and length of stay prediction.

## Datasets

- **MIMIC-IV 2.2** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUC-ROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
- `PR-AUC` — range: [0, 1]
  - Area under the Precision-Recall curve. Measures the trade-off between precision and recall across all classification thresholds, particularly useful for imbalanced clinical datasets.

## Input / output format

**Input**: 2-hour resolution time series of lab/vital measurements and ICD-10 diagnosis codes for the first 48 hours of an ICU admission.

**Output**: Binary classification label: 1 for in-ICU mortality or length of stay ≥ 3 days, 0 otherwise.

## Scoring recipe

```python
def compute_auc_roc(y_true, y_pred):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

def compute_pr_auc(y_true, y_pred):
    precision, recall, _ = precision_recall_curve(y_true, y_pred)
    return auc(recall, precision)

# 5-fold CV: 80% train, 20% test. 10% of train used for validation.
# Average metrics across all folds.
```

## Common pitfalls

- The dataset uses forward-fill and mean imputation for missing labs/vitals, which can artificially inflate performance if models assume regular sampling.
- Table 3 incorrectly labels the second metric as 'PR-ROC' instead of PR-AUC, which may cause confusion when reproducing results.
- Evaluation is strictly limited to ICU stays for Chronic Kidney Disease patients, so results do not generalize to other wards or conditions.

## Evidence (verbatim from paper)

> Regarding evaluation, we utilize area under the ROC curve (OC-AUC score) and area under precision-recall curve (PR-AUC score). ... We use 5-folded cross validation to verify the model performance, in which the dataset is spit 80% for training, and 20% for testing.

## Citation

```bibtex
@misc{bui2024benchmarking,
  title={Benchmarking with MIMIC-IV, an irregular, spare clinical time series dataset},
  author={Bui et al. (2024)},
  year={2024},
  note={arXiv:2401.15290}
}
```

- arXiv: 2401.15290