Mimic Iii Healthcare Benchmark Eval

---
name: mimic-iii-healthcare-benchmark-eval
description: This benchmark evaluates deep learning and traditional machine learning models on critical care prediction tasks using the MIMIC-III dataset. It probes a model's ability to predict patient mortality across multiple time horizons, classify ICD-9 diagnosis groups, and forecast hospital length of stay from raw clinical time series and tabular data. Use when the user wants to benchmark on MIMIC-III, or asks about evaluating this task. Reports binary classification.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1710.08531
  bibtex_key: purushotham2017benchmark
  confidence: high
---

# mimic-iii-healthcare-benchmark-eval

> Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets — Purushotham et al. (2017) (arXiv:1710.08531, 2017)

## What this evaluates

This benchmark evaluates deep learning and traditional machine learning models on critical care prediction tasks using the MIMIC-III dataset. It probes a model's ability to predict patient mortality across multiple time horizons, classify ICD-9 diagnosis groups, and forecast hospital length of stay from raw clinical time series and tabular data.

## Datasets

- **MIMIC-III** — total 35637; splits: train (-1), val (-1), test (-1)

## Metrics

- `binary classification` **(primary)** — range: [0, 1]
  - Standard classification metrics (e.g., AUC-ROC, accuracy) computed on predicted probabilities or hard labels versus ground truth death events or ICD-9 diagnosis groups.
- `regression` — range: other
  - Standard regression metrics (e.g., RMSE, MAE) computed on predicted continuous duration versus actual hospital length of stay in hours.

## Input / output format

**Input**: Raw clinical time series data (vital signs, lab results) from the first 24 or 48 hours of ICU admission, combined with non-temporal patient demographics and clinical features.

**Output**: For mortality: binary label (1 for death, 0 for survival). For ICD-9: 20-class diagnosis group labels. For length of stay: continuous float representing duration in hours.

## Scoring recipe

```python
def score_classification(y_true, y_pred_proba):
    auc = roc_auc_score(y_true, y_pred_proba)
    acc = accuracy_score(y_true, (y_pred_proba >= 0.5).astype(int))
    return {'AUC-ROC': auc, 'Accuracy': acc}

def score_regression(y_true, y_pred):
    rmse = np.sqrt(mean_squared_error(y_true, y_pred))
    mae = mean_absolute_error(y_true, y_pred)
    return {'RMSE': rmse, 'MAE': mae}
```

## Common pitfalls

- Failing to use patient-level data splitting, which causes data leakage when multiple admissions from the same patient appear in both train and test sets.
- Applying heavy feature engineering or preprocessing to the time series, which contradicts the benchmark's goal of evaluating end-to-end deep learning on raw clinical data.
- Ignoring severe class imbalance in short-term mortality labels (e.g., ~1.4% positive rate), leading to misleading accuracy scores without proper handling (e.g., F1-score or AUC-ROC).

## Evidence (verbatim from paper)

> We formulate mortality as a binary classification task, where the label indicates the death event for a patient. ... We treat length of stay prediction task as a regression problem.

## Citation

```bibtex
@misc{purushotham2017benchmark,
  title={Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets},
  author={Purushotham et al. (2017)},
  year={2017},
  note={arXiv:1710.08531}
}
```

- arXiv: 1710.08531