Critical Icu Prediction Eval

---
name: critical-icu-prediction-eval
description: Evaluates traditional machine learning and deep learning models on a large-scale, multi-institutional OMOP CDM dataset for ICU clinical prediction. It probes the ability of models to forecast patient outcomes (mortality, length of stay, readmission, sepsis) using early admission temporal features. Use when the user wants to benchmark on CRITICAL, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.08247
  bibtex_key: luo2025crisp
  confidence: high
---

# critical-icu-prediction-eval

> The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Data — Luo et al. (2025) (arXiv:2509.08247, 2025)

## What this evaluates

Evaluates traditional machine learning and deep learning models on a large-scale, multi-institutional OMOP CDM dataset for ICU clinical prediction. It probes the ability of models to forecast patient outcomes (mortality, length of stay, readmission, sepsis) using early admission temporal features.

## Datasets

- **CRITICAL** — total 1950000000; splits: train (-1), test (-1); repo https://github.com/AaronLuo00/CRISP-Pipeline

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds. Values range from 0.5 (random guessing) to 1.0 (perfect discrimination).

## Input / output format

**Input**: 800 high-frequency clinical features extracted from five OMOP CDM tables (MEASUREMENT, OBSERVATION, DRUG_EXPOSURE, CONDITION_OCCURRENCE, PROCEDURE_OCCURRENCE), discretized into 4-hour bins over a 24-hour observation window (or 48-hour window for mortality/readmission tasks).

**Output**: Binary classification label indicating the presence or absence of a specific ICU outcome (7-day/30-day mortality, LOS >3 or >7 days, 7/30/90-day readmission, or sepsis onset within 48h/7 days).

## Scoring recipe

```python
def compute_auroc(y_true, y_pred):
    # y_true: binary ground truth labels (0 or 1)
    # y_pred: predicted probabilities for the positive class
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

# Evaluation uses 5-fold cross-validation with 80% train / 20% test split.
# Final metric is the mean AUROC across folds.
```

## Common pitfalls

- A strict 48-hour gap must be maintained between the observation window and the prediction window to prevent label leakage.
- Features are strictly limited to the 800 most frequent clinical concepts across five specific OMOP tables, not the full harmonized dataset.
- Performance is reported via 5-fold cross-validation with an 80/20 train/test split, not a single fixed held-out test set.

## Evidence (verbatim from paper)

> All models employ 5-fold cross-validation, with the final results reported using 80% of data for training and 20% for testing. ... Table 1: Clinical Prediction Performance (AUROC) ... Even with CRISP’s harmonization and sparsity reduction, the best models achieve only 0.619-0.755 AUROC for readmission tasks, highlighting the inherent complexity of predicting patient trajectories across heterogeneous institutions.

## Citation

```bibtex
@misc{luo2025crisp,
  title={The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Data},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2509.08247}
}
```

- arXiv: 2509.08247