Ecg Multitask Eval

---
name: ecg-multitask-eval
description: Evaluates the ability of foundation models (LLMs, time-series, and ECG-specific) and traditional deep learning models to perform regression and classification tasks on electrocardiogram (ECG) signals across zero-shot, few-shot, and fine-tuned settings. Use when the user wants to benchmark on ECG Multi-task Benchmark, or asks about evaluating this task. Reports MAE, F1 Score, Accuracy (ACC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.08954
  bibtex_key: xu2025ecgmultitask
  confidence: high
---

# ecg-multitask-eval

> An Electrocardiogram Multi-task Benchmark with Comprehensive Evaluations and Insightful Findings — Xu et al. (2025) (arXiv:2512.08954, 2025)

## What this evaluates

Evaluates the ability of foundation models (LLMs, time-series, and ECG-specific) and traditional deep learning models to perform regression and classification tasks on electrocardiogram (ECG) signals across zero-shot, few-shot, and fine-tuned settings.

## Datasets

- **ECG Multi-task Benchmark** — total ?; splits: test (-1)

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: the average of the absolute differences between predicted and true values. Lower is better.
- `F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Higher is better.
- `Accuracy (ACC)` **(primary)** — range: [0, 1]
  - Proportion of correct predictions out of total predictions: sum(pred == true) / len(true). Higher is better.
- `Win Rate` — range: percent
  - Percentage of tasks or settings where a model achieves the top score compared to baselines. Higher is better.

## Input / output format

**Input**: ECG signal segments (preprocessed or downsampled to match model input lengths), optionally accompanied by task-specific prompts or labels for supervised/fine-tuning settings.

**Output**: Predicted continuous values for regression tasks (RR interval, age), predicted class labels for classification tasks (gender, potassium abnormality, arrhythmia subtype), or saliency maps for interpretability.

## Scoring recipe

```python
def score(predictions, gold, task):
    if task in ['RR_interval', 'Age']:
        return mean(abs(predictions - gold))  # MAE
    elif task in ['Gender', 'Potassium']:
        tp = sum((predictions == 1) & (gold == 1))
        fp = sum((predictions == 1) & (gold == 0))
        fn = sum((predictions == 0) & (gold == 1))
        prec = tp / max(tp + fp, 1)
        rec = tp / max(tp + fn, 1)
        return 2 * prec * rec / (prec + rec)  # F1
    elif task == 'Arrhythmia':
        return sum(predictions == gold) / len(gold)  # ACC
    elif task == 'Benchmark':
        wins = sum(1 for m in predictions if m == max(predictions))
        return wins / len(predictions)  # Win Rate
```

## Common pitfalls

- LLMs struggle with raw time-series ECG data without extensive prompt engineering or feature extraction, often performing worse than traditional time-series models.
- Zero-shot and few-shot settings frequently yield suboptimal results for foundation models due to domain shifts between pretraining corpora and clinical ECG distributions.
- Fine-tuning requires sufficient labeled samples; limited tuning can hinder effective adaptation to ECG-specific tasks.

## Evidence (verbatim from paper)

> Table 2: Benchmarking experimental results. Highlighted are the top first, second, and third results. (RR Interval Estimation, Age Estimation, Gender Classification, Potassium Abnormality Prediction, Arrhythmia Detection, and zero-shot, few-shot, fine-tune are denoted as RR., Age, Gen., Ka, AD, and zs, fs, ft respectively.) Regre. (MAE)↓ | Binary Class (F1 Score)↑ | 15 Class (ACC)↑ | Benchmark (Win Rate)↑

## Citation

```bibtex
@misc{xu2025ecgmultitask,
  title={An Electrocardiogram Multi-task Benchmark with Comprehensive Evaluations and Insightful Findings},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2512.08954}
}
```

- arXiv: 2512.08954