Ecg Multi Label Eval

---
name: ecg-multi-label-eval
description: This evaluation probes the ability of ECG foundation models to learn robust, generalizable representations from unsupervised pretraining and transfer them to downstream multi-label classification tasks. It specifically tests generalization across different clinical datasets and sampling rates by measuring performance on arrhythmia conditions and rhythm classifications. Use when the user wants to benchmark on PTB-XL, Chapman, or asks about evaluating this task. Reports macro AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.00102
  bibtex_key: nguyen2025ecgsoup
  confidence: high
---

# ecg-multi-label-eval

> ECG-Soup: Harnessing Multi-Layer Synergy for ECG Foundation Models — Phu X. Nguyen et al. (2025) (arXiv:2509.00102, 2025)

## What this evaluates

This evaluation probes the ability of ECG foundation models to learn robust, generalizable representations from unsupervised pretraining and transfer them to downstream multi-label classification tasks. It specifically tests generalization across different clinical datasets and sampling rates by measuring performance on arrhythmia conditions and rhythm classifications.

## Datasets

- **PTB-XL** — total 21837; splits: train (-1), val (-1), test (-1)
- **Chapman** — total 10646; splits: train (-1), val (-1), test (-1)

## Metrics

- `macro AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve computed per class and averaged across all classes (macro average).
- `macro F1-score` — range: [0, 1]
  - F1-score computed per class and averaged across all classes (macro average).
- `sample accuracy` — range: [0, 1]
  - Accuracy computed per sample (instance) and then averaged across all samples.

## Input / output format

**Input**: 12-lead ECG signals normalized to a fixed 100 Hz sampling rate and 10-second duration (truncated or zero-padded).

**Output**: Binary probability scores for each of the 71 PTB-XL conditions, 67 Chapman conditions, 12 PTB-XL rhythms, or 11 Chapman rhythms.

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import average_precision_score, f1_score, accuracy_score

def evaluate(y_true, y_pred):
    # y_true: (N, C) binary ground truth
    # y_pred: (N, C) predicted probabilities
    macro_auc = average_precision_score(y_true, y_pred, average='macro')
    macro_f1 = f1_score(y_true, (y_pred > 0.5).astype(int), average='macro')
    sample_acc = accuracy_score(y_true.flatten(), (y_pred > 0.5).astype(int).flatten())
    return {'macro_auc': macro_auc, 'macro_f1': macro_f1, 'sample_acc': sample_acc}
```

## Common pitfalls

- Data leakage between pretraining and evaluation: CinC2020 contains PTB-XL recordings. The OOD scenario explicitly removes PTB-XL from pretraining, but in-distribution evaluation retains it, requiring careful dataset curation.
- Signal preprocessing mismatch: All inputs must be resampled to 100 Hz and padded/truncated to exactly 10 seconds. Deviations break the fixed-length ViT patching mechanism.
- Multi-label metric aggregation: Metrics are reported as both macro (per-label averaged) and sample (per-sample averaged). Confusing these aggregation methods leads to incorrect benchmarking.

## Evidence (verbatim from paper)

> Model performance was assessed using both macro- and sample-level metrics, including macro/sample AUC, instance/sample accuracy, and macro/sample F1-score, providing a comprehensive evaluation of the models in multi-label classification settings.

## Citation

```bibtex
@misc{nguyen2025ecgsoup,
  title={ECG-Soup: Harnessing Multi-Layer Synergy for ECG Foundation Models},
  author={Phu X. Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2509.00102}
}
```

- arXiv: 2509.00102