Ecg Language Models Eval

---
name: ecg-language-models-eval
description: Evaluates the ability of encoder-free ECG-language models to process raw ECG signals alongside textual queries for medical question answering and instruction following. Probes whether models genuinely leverage physiological ECG data or rely on language priors and benchmark artifacts. Use when the user wants to benchmark on PTB-XL ECG-QA, PULSE ECG-Bench, ECG-Chat Instruct, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.18798
  bibtex_key: han2026encoderfreeecglanguagemodels
  confidence: high
---

# ecg-language-models-eval

> Encoder-Free ECG-Language Models — William Han et al. (2026) (arXiv:2601.18798, 2026)

## What this evaluates

Evaluates the ability of encoder-free ECG-language models to process raw ECG signals alongside textual queries for medical question answering and instruction following. Probes whether models genuinely leverage physiological ECG data or rely on language priors and benchmark artifacts.

## Datasets

- **PTB-XL ECG-QA** — total ?; splits: test (-1)
- **PULSE ECG-Bench** — total ?; splits: test (-1)
- **ECG-Chat Instruct** — total ?; splits: test (-1)

## Metrics

- `BLEU-4` — range: other
  - Computes the geometric mean of modified n-gram precisions up to 4-grams, typically scaled to [0, 100] in this work.
- `Accuracy` **(primary)** — range: percent
  - Percentage of instances where the model's generated response exactly matches the ground truth answer.

## Input / output format

**Input**: ECG signal (1D tensor) or ECG image paired with a textual query/question.

**Output**: Textual response to the query.

## Scoring recipe

```python
def evaluate(predictions, golds):
    bleu4 = compute_bleu_4(predictions, golds)
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    return {"BLEU-4": bleu4, "Accuracy": acc}
```

## Common pitfalls

- Models may achieve high accuracy by relying on language priors rather than actual ECG signal content, as shown by comparable performance when ECG input is replaced with zeros or omitted.
- BLEU-4 scores can be inflated by generic medical phrasing in responses that do not accurately reflect the specific ECG data provided.
- Evaluating with 'Only Text' or 'Zeros Tensor' inputs reveals that benchmark accuracy may not meaningfully depend on the physiological modality.

## Evidence (verbatim from paper)

> Table 1 reports the BLEU-4 and accuracy scores when training and evaluating on the PULSE ECG-Instruct and PULSE ECG-Bench datasets respectively. We report mean ± standard deviations over three seeds evaluated on PTB-XL ECG-QA.

## Citation

```bibtex
@misc{han2026encoderfreeecglanguagemodels,
  title={Encoder-Free ECG-Language Models},
  author={William Han et al. (2026)},
  year={2026},
  note={arXiv:2601.18798}
}
```

- arXiv: 2601.18798