Climb Eval

---
name: climb-eval
description: Evaluates clinical foundation models across diverse medical modalities (imaging, time series, graphs, text) using multitask pretraining, few-shot transfer, and multimodal fusion. It probes model robustness on understudied tasks, adaptation to limited labeled data, and integration of heterogeneous clinical signals for prognosis. Use when the user wants to benchmark on CLIMB, or asks about evaluating this task. Reports balanced AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.07667
  bibtex_key: dai2025climb
  confidence: high
---

# climb-eval

> CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models — Dai et al. (2025) (arXiv:2503.07667, 2025)

## What this evaluates

Evaluates clinical foundation models across diverse medical modalities (imaging, time series, graphs, text) using multitask pretraining, few-shot transfer, and multimodal fusion. It probes model robustness on understudied tasks, adaptation to limited labeled data, and integration of heterogeneous clinical signals for prognosis.

## Datasets

- **CLIMB** — total 4510000; splits: train (-1), val (-1), test (-1); repo https://github.com/DDVD233/climb

## Metrics

- `balanced AUC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve, computed with class balancing to ensure equal weight to positive and negative classes.
- `sensitivity` — range: [0, 1]
  - True positive rate (TP / (TP + FN)), measuring the proportion of actual positives correctly identified.
- `specificity` — range: [0, 1]
  - True negative rate (TN / (TN + FP)), measuring the proportion of actual negatives correctly identified.
- `MAE` — range: [0, inf)
  - Mean absolute error: average of absolute differences between predicted and actual values, used for regression tasks like length of stay prediction.

## Input / output format

**Input**: Multimodal clinical data including 2D/3D medical images, video, time series (ECG, EEG), graph structures, and clinical text. Inputs are paired with task-specific labels (diagnostic categories, LOS values, or binary mortality outcomes).

**Output**: Task-specific predictions: diagnostic class labels for classification tasks, continuous values for length-of-stay regression, or binary probabilities for 48-hour in-hospital mortality.

## Scoring recipe

```python
def score(predictions, golds, task_type):
    if task_type == 'classification':
        auc = roc_auc_score(golds, predictions, average='balanced')
        fpr, tpr, _ = roc_curve(golds, predictions)
        idx = np.argmax(tpr - fpr)
        return {'balanced_AUC': auc, 'sensitivity': tpr[idx], 'specificity': 1 - fpr[idx]}
    elif task_type == 'regression':
        return {'MAE': mean_absolute_error(golds, predictions)}
```

## Common pitfalls

- Using standard AUC instead of balanced AUC can mask poor performance on minority clinical classes.
- Assuming complex fusion strategies (e.g., cross-attention) always outperform simpler ones; the paper shows MLP fusion suffices for simpler tasks like 48h mortality.
- Evaluating general-domain encoders without clinical fine-tuning may yield misleading baselines, as performance varies heavily by modality (e.g., ConvNeXTv2 excels in vision but specialized models win in ECG).

## Evidence (verbatim from paper)

> Evaluation metrics. For consistency, we evaluate all classification tasks with balanced AUC, sensitivity, and specificity. Regression tasks (e.g., length of stay prediction) are evaluated with mean absolute error (MAE).

## Citation

```bibtex
@misc{dai2025climb,
  title={CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models},
  author={Dai et al. (2025)},
  year={2025},
  note={arXiv:2503.07667}
}
```

- arXiv: 2503.07667