Ecg Benchmark Eval

---
name: ecg-benchmark-eval
description: Evaluates the capability of models to analyze electrocardiogram (ECG) time-series data across four medical tasks: classification, detection, forecasting, and generation. It probes semantic fidelity and diagnostic accuracy in quasi-periodic physiological signals, emphasizing robustness to temporal shifts and class imbalance. Use when the user wants to benchmark on CPSC2018, CPSC2019, CPSC2020, CPSC2021, MITDB, PTBXL, FEPL, DALIA, SST, or asks about evaluating this task. Reports FFD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.14206
  bibtex_key: tang2025ecgbenchmark
  confidence: high
---

# ecg-benchmark-eval

> A Comprehensive Benchmark for Electrocardiogram Time-Series — Tang et al. (2025) (arXiv:2507.14206, 2025)

## What this evaluates

Evaluates the capability of models to analyze electrocardiogram (ECG) time-series data across four medical tasks: classification, detection, forecasting, and generation. It probes semantic fidelity and diagnostic accuracy in quasi-periodic physiological signals, emphasizing robustness to temporal shifts and class imbalance.

## Datasets

- **CPSC2018** — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
- **CPSC2019** — total 2000; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
- **CPSC2020** — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
- **CPSC2021** — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
- **MITDB** — total 48; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
- **PTBXL** — total 21799; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
- **FEPL** — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
- **DALIA** — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
- **SST** — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark

## Metrics

- `accuracy` — range: [0, 1]
  - Proportion of correctly classified ECG recordings out of the total test set. Computed as correct predictions divided by total samples.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for detection tasks. Predictions are matched to ground truth if within a ±70ms temporal window.
- `FFD` **(primary)** — range: [0, inf)
  - Feature-based Fréchet Distance. Computes the Fréchet distance between the latent feature distributions of real and predicted/generated ECG sequences, extracted via a transformer encoder mapping. Lower values indicate higher semantic fidelity.

## Input / output format

**Input**: Single-channel ECG time-series recordings resampled to 100 Hz with a fixed length of 500 points. Multi-channel recordings are split into single-channel. Missing data (>25%) is discarded; otherwise, linear interpolation is used.

**Output**: Classification: discrete disease/arrhythmia class labels. Detection: predicted waveform probability scores with temporal positions. Forecasting: synthetic ECG time-series sequences of 100 points. Generation: full-length synthetic ECG sequences.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task):
    if task == 'classification':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task == 'detection':
        tp = sum(1 for p, g in zip(predictions, golds) if abs(p - g) <= 0.070)
        fp = len(predictions) - tp
        fn = len(golds) - tp
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif task in ('forecasting', 'generation'):
        real_feats = [transformer_encoder(x) for x in golds]
        pred_feats = [transformer_encoder(x) for x in predictions]
        return frechet_distance(real_feats, pred_feats)
```

## Common pitfalls

- Using Mean Squared Error (MSE) instead of FFD for generation/forecasting, as MSE is highly sensitive to minor temporal shifts and fails to capture clinical semantics in quasi-periodic ECG signals.
- Ignoring the ±70ms matching window for detection tasks, which leads to artificially low recall and F1 scores despite clinically acceptable predictions.
- Failing to address severe class imbalance in detection tasks without adopting F1-score or proper downsampling/oversampling strategies.

## Evidence (verbatim from paper)

> Our proposed method, PSSM, achieves state-of-the-art performance across all tasks, with an average performance of 0.947 in classification accuracy, 0.820 in detection F1 score, 0.211 in forecasting FFD, and 0.133 in generation FFD.

## Citation

```bibtex
@misc{tang2025ecgbenchmark,
  title={A Comprehensive Benchmark for Electrocardiogram Time-Series},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2507.14206}
}
```

- arXiv: 2507.14206