Hirid Icu Benchmark Eval

---
name: hirid-icu-benchmark-eval
description: Evaluates machine learning and deep learning models on predicting clinical outcomes from high-resolution ICU time-series data. It probes capabilities in handling class imbalance, long temporal dependencies, and varying data resolutions across stay-level and online monitoring tasks. Use when the user wants to benchmark on HiRID, or asks about evaluating this task. Reports AUPRC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2111.08536
  bibtex_key: yeche2021hirid
  confidence: high
---

# hirid-icu-benchmark-eval

> HiRID-ICU-Benchmark -- A Comprehensive Machine Learning Benchmark on High-resolution ICU Data — Yèche et al. (2021) (arXiv:2111.08536, 2021)

## What this evaluates

Evaluates machine learning and deep learning models on predicting clinical outcomes from high-resolution ICU time-series data. It probes capabilities in handling class imbalance, long temporal dependencies, and varying data resolutions across stay-level and online monitoring tasks.

## Datasets

- **HiRID** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/ratschlab/HIRID-ICU-Benchmark

## Metrics

- `AUPRC` **(primary)** — range: [0, 1]
  - Area under the Precision-Recall Curve. Computed by integrating precision over recall thresholds across all classification thresholds.
- `AUROC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic Curve. Measures the trade-off between true positive rate and false positive rate across thresholds.
- `B-Accuracy` — range: [0, 1]
  - Balanced Accuracy, calculated as the average of sensitivity (recall) and specificity across classes.
- `MAE` — range: other
  - Mean Absolute Error, the average of absolute differences between predicted and true continuous values.

## Input / output format

**Input**: High-resolution ICU time-series data sampled at 5-minute intervals. Deep learning models receive the full historical sequence (up to 2016 steps/1 week), while traditional ML models receive only the current time-step features.

**Output**: Task-specific predictions: binary/multi-class labels for ICU Mortality, Patient Phenotyping, Circulatory/Respiratory Failure; continuous values for Kidney Function (ml/kg/h) and Remaining LOS (hours).

## Scoring recipe

```python
def evaluate(predictions, gold, task):
    scores = {}
    if task in ['ICU Mortality', 'Circulatory Failure', 'Respiratory Failure']:
        scores['AUPRC'] = average_precision_score(gold, predictions)
        scores['AUROC'] = roc_auc_score(gold, predictions)
    elif task == 'Patient Phenotyping':
        scores['B-Accuracy'] = balanced_accuracy_score(gold, (predictions >= 0.5).astype(int))
    elif task in ['Kidney Function', 'Remaining LOS']:
        scores['MAE'] = mean_absolute_error(gold, predictions)
    return scores
# Final reported metric is mean ± std over 10 independent runs with different random seeds.
```

## Common pitfalls

- Applying balanced loss weights to highly imbalanced binary tasks (e.g., ICU Mortality, Circulatory Failure) degrades performance, contrary to common practice.
- Deep learning models do not automatically benefit from longer history; performance on online tasks often plateaus or drops when sequence length exceeds 12 hours due to architectural limitations.
- Reporting metrics scaled to 100 for readability can cause confusion if not explicitly noted as percentages.

## Evidence (verbatim from paper)

> For all models, we tuned specific hyper-parameters using random search. Each randomly picked set of parameters was run with 3 different random initializations. We then selected hyper-parameters on the validation set performance for either AUPRC, B-Accuracy, or MAE.

## Citation

```bibtex
@misc{yeche2021hirid,
  title={HiRID-ICU-Benchmark -- A Comprehensive Machine Learning Benchmark on High-resolution ICU Data},
  author={Yèche et al. (2021)},
  year={2021},
  note={arXiv:2111.08536}
}
```

- arXiv: 2111.08536