Ecg Fm Benchmark Eval

---
name: ecg-fm-benchmark-eval
description: Evaluates the clinical utility and label efficiency of ECG foundation models across diverse tasks including adult/pediatric ECG interpretation, cardiac structure prediction, clinical outcome forecasting, and patient characteristic regression. It probes cross-domain generalization, fine-tuning adaptability, and the quality of frozen/linear representations compared to strong supervised baselines. Use when the user wants to benchmark on PTB-XL, EchoNext, MIMIC-IV (ECG), CPSC2018, PTB, Ningbo, Georgia, Chapman, SPH, CODE-15%, ZZU pECG, or asks about evaluating this task. Reports macro-AUROC, average z-normalized MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.25095
  bibtex_key: almasud2025ecgfmbenchmarking
  confidence: high
---

# ecg-fm-benchmark-eval

> Benchmarking ECG Foundational Models: A Reality Check Across Clinical Tasks — Al-Masud et al. (2025) (arXiv:2509.25095, 2025)

## What this evaluates

Evaluates the clinical utility and label efficiency of ECG foundation models across diverse tasks including adult/pediatric ECG interpretation, cardiac structure prediction, clinical outcome forecasting, and patient characteristic regression. It probes cross-domain generalization, fine-tuning adaptability, and the quality of frozen/linear representations compared to strong supervised baselines.

## Datasets

- **PTB-XL** — total ?; splits: test (-1)
- **EchoNext** — total ?; splits: test (-1)
- **MIMIC-IV (ECG)** — total ?; splits: test (-1)
- **CPSC2018** — total ?; splits: test (-1)
- **PTB** — total ?; splits: test (-1)
- **Ningbo** — total ?; splits: test (-1)
- **Georgia** — total ?; splits: test (-1)
- **Chapman** — total ?; splits: test (-1)
- **SPH** — total ?; splits: test (-1)
- **CODE-15%** — total ?; splits: test (-1)
- **ZZU pECG** — total ?; splits: test (-1)

## Metrics

- `macro-AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, averaged across all classes (macro-averaged). Higher values indicate better classification performance.
- `average z-normalized MAE` **(primary)** — range: [0, inf)
  - Mean Absolute Error computed on z-normalized target and prediction vectors: MAE(z(y_true), z(y_pred)). Normalization mitigates scale differences across regression targets.
- `label efficiency ratio` — range: [0, 1]
  - Ratio r = N*/N, where N* is the training set size required for a pretrained model to match the performance of a supervised baseline trained on N samples. Derived from parametric scaling curve fits.

## Input / output format

**Input**: ECG time-series signals (typically 12-lead), optionally with patient metadata for regression tasks.

**Output**: Class labels for interpretation/outcome tasks; continuous values for biometric/clinical regression tasks.

## Scoring recipe

```python
def macro_auroc(y_true, y_pred):
    scores = [roc_auc_score(y_true[:, i], y_pred[:, i]) for i in range(y_true.shape[1])]
    return np.mean(scores)

def z_norm_mae(y_true, y_pred):
    z_true = (y_true - np.mean(y_true)) / np.std(y_true)
    z_pred = (y_pred - np.mean(y_pred)) / np.std(y_pred)
    return np.mean(np.abs(z_true - z_pred))
```

## Common pitfalls

- Comparing frozen/linear evaluation directly with finetuning without acknowledging that they measure different aspects of representational quality.
- Using raw MAE instead of z-normalized MAE for regression tasks, which unfairly penalizes models on targets with high variance or different scales.
- Ignoring stratified subsampling protocols when evaluating label efficiency, leading to biased scaling curves.

## Evidence (verbatim from paper)

> Table 2: Comparison of macro-AUROC (classification) and average z-normalized MAE (regression) under finetuning with linear prediction head. ... We use the parametric form of the fits to work out a label efficiency ratio r=N*/N, i.e., the fraction of samples N* required for the pretrained model to reach the same performance as the supervised baseline for given N.

## Citation

```bibtex
@misc{almasud2025ecgfmbenchmarking,
  title={Benchmarking ECG Foundational Models: A Reality Check Across Clinical Tasks},
  author={Al-Masud et al. (2025)},
  year={2025},
  note={arXiv:2509.25095}
}
```

- arXiv: 2509.25095