Carebench Eval

---
name: carebench-eval
description: Evaluates multimodal fusion of Electronic Health Records (EHR) and Chest X-Rays (CXR) for clinical decision support, specifically testing robustness to missing modalities, temporal imbalance, and subgroup fairness across phenotyping, mortality, and length-of-stay prediction. Use when the user wants to benchmark on CareBench, or asks about evaluating this task. Reports AUROC, AUPRC, F1, Accuracy, Cohen's Kappa.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.23614
  bibtex_key: yin2026carebench
  confidence: high
---

# carebench-eval

> When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion — Yin et al. (2026) (arXiv:2602.23614, 2026)

## What this evaluates

Evaluates multimodal fusion of Electronic Health Records (EHR) and Chest X-Rays (CXR) for clinical decision support, specifically testing robustness to missing modalities, temporal imbalance, and subgroup fairness across phenotyping, mortality, and length-of-stay prediction.

## Datasets

- **CareBench** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/jakeykj/CareBench

## Metrics

- `AUROC, AUPRC, F1, Accuracy, Cohen's Kappa` **(primary)** — range: [0, 1]
  - AUROC measures threshold-independent discrimination; AUPRC focuses on positive class performance; F1 is the harmonic mean of precision and recall; Accuracy is the proportion of correct predictions; Cohen's Kappa weighted quadratic penalizes ordinal misclassifications proportionally to their distance.

## Input / output format

**Input**: Patient-level EHR time-series data and corresponding Chest X-Ray images, restricted to a fixed prediction window (e.g., first 48 hours of ICU stay).

**Output**: Multi-label binary vector (25 phenotypes), binary mortality prediction (survive/die), or multi-class ordinal label for remaining length of stay (RLOS: 2–3, 3–4, 4–5, 5–6, 6–7, 7–14, 14+ days).

## Scoring recipe

```python
def calc_metrics(y_true, y_pred, y_prob=None):
    if y_prob is None: y_prob = y_pred
    metrics = {
        'AUROC': roc_auc_score(y_true, y_prob, average='macro'),
        'AUPRC': average_precision_score(y_true, y_prob, average='macro'),
        'F1': f1_score(y_true, y_pred, average='macro'),
        'ACC': accuracy_score(y_true, y_pred)
    }
    if task == 'LoS':
        metrics['Kappa'] = cohen_kappa_score(y_true, y_pred, weights='quadratic')
    return metrics
```

## Common pitfalls

- Evaluating only on the matched subset (complete modalities) overestimates real-world performance; the base cohort with missing modalities must be used for clinical relevance.
- Temporal imbalance in EHR data and modality missingness rapidly degrade multimodal gains, which is often overlooked in standard benchmarks.
- Multimodal fusion does not automatically improve algorithmic fairness; subgroup disparities depend on demographic sensitivity differences rather than modality count.

## Evidence (verbatim from paper)

> To ensure comparability, we adopt patient-level train/validation/test splits and report established metrics tailored to each task. ... we employ a suite of metrics including Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Precision-Recall Curve (AUPRC), F1 score, precision, recall, specificity, and accuracy (ACC). ... Performance is evaluated using the ACC, F1 score, and Cohen’s Kappa weighted quadratic

## Citation

```bibtex
@misc{yin2026carebench,
  title={When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion},
  author={Yin et al. (2026)},
  year={2026},
  note={arXiv:2602.23614}
}
```

- arXiv: 2602.23614