Benchmd Eval

---
name: benchmd-eval
description: Evaluates modality-agnostic models across 19 real-world medical datasets spanning 1D, 2D, and 3D modalities. Probes performance under data scarcity (few-shot linear evaluation and finetuning) and out-of-distribution generalization across different hospitals and data distributions. Use when the user wants to benchmark on BenchMD, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.08486
  bibtex_key: [REDACTED]
  confidence: high
---

# benchmd-eval

> BenchMD: A Benchmark for Unified Learning on Medical Images and Sensors — Wantlin et al. (2023) (arXiv:2304.08486, 2023)

## What this evaluates

Evaluates modality-agnostic models across 19 real-world medical datasets spanning 1D, 2D, and 3D modalities. Probes performance under data scarcity (few-shot linear evaluation and finetuning) and out-of-distribution generalization across different hospitals and data distributions.

## Datasets

- **BenchMD** — total 19; splits: train (-1), val (-1), test (-1); repo https://github.com/rajpurkarlab/BenchMD

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve. For multi-class tasks, computed as the unweighted average of per-class AUROC scores. For multi-label tasks, similarly averaged across classes.

## Input / output format

**Input**: Raw medical data (1D sensor signals, 2D images, or 3D volumes) processed through modality-specific embedding modules into a unified 256-dimensional space. Models are trained on labeled source data and evaluated zero-shot on unlabeled OOD target data.

**Output**: Class probabilities or logits from a linear classifier head or finetuned backbone, used to compute per-class AUROC scores.

## Scoring recipe

```python
def compute_auroc(y_true, y_pred, task_type='single'):
    scores = []
    classes = np.unique(y_true)
    for c in classes:
        y_bin = (y_true == c).astype(int)
        scores.append(roc_auc_score(y_bin, y_pred[:, c]))
    return np.mean(scores)
```

## Common pitfalls

- Performance is highly modality-dependent; no single SSL or pretraining method dominates across all 19 datasets.
- In-distribution validation performance does not reliably predict out-of-distribution test performance due to distribution shifts across hospitals.
- Label availability settings (8, 64, 256 per class) are applied differently for single-label vs multi-label tasks, which can cause overfitting or performance drops on OOD data.

## Evidence (verbatim from paper)

> We evaluate our models using AUROC score as the metric (taking an unweighted average of per-class scores for multi-class tasks).

## Citation

```bibtex
@misc{wantlin2023benchmd,
  title={BenchMD: A Benchmark for Unified Learning on Medical Images and Sensors},
  author={Wantlin et al. (2023)},
  year={2023},
  note={arXiv:2304.08486}
}
```

- arXiv: 2304.08486