Camel Ecg Eval

---
name: camel-ecg-eval
description: Evaluates a multimodal language model's ability to process long-duration ECG signals for clinical forecasting, diagnostic classification, report generation, and statistical grounding. It probes temporal reasoning, multi-lead interpretation, and instruction-following in medical AI settings. Use when the user wants to benchmark on Icentia11k, PTB-XL, CSN, CODE-15%, CPSC-2018, HEEDB, Penn, MIMIC-IV-ECG, ECGBench / ECG-QA, ECG Grounding Benchmark, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.15677
  bibtex_key: velingker2026camel
  confidence: high
---

# camel-ecg-eval

> CAMEL: An ECG Language Model for Forecasting Cardiac Events — Velingker et al. (2026) (arXiv:2602.15677, 2026)

## What this evaluates

Evaluates a multimodal language model's ability to process long-duration ECG signals for clinical forecasting, diagnostic classification, report generation, and statistical grounding. It probes temporal reasoning, multi-lead interpretation, and instruction-following in medical AI settings.

## Datasets

- **Icentia11k** — total ?; splits: test (-1)
- **PTB-XL** — total ?; splits: test (-1)
- **CSN** — total ?; splits: test (-1)
- **CODE-15%** — total ?; splits: test (-1)
- **CPSC-2018** — total ?; splits: test (-1)
- **HEEDB** — total ?; splits: test (-1)
- **Penn** — total ?; splits: test (-1)
- **MIMIC-IV-ECG** — total ?; splits: test (-1)
- **ECGBench / ECG-QA** — total ?; splits: test (-1)
- **ECG Grounding Benchmark** — total 2000; splits: test (2000)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall. Reported as macro-average across classes or forecasting horizons.
- `AUROC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive and false positive rates for linear probing.
- `RMSE` — range: other
  - Root Mean Squared Error between predicted and ground-truth ECG statistics (e.g., RR interval, HR, QRS duration).
- `Accuracy` — range: [0, 1]
  - Fraction of correctly predicted multiple-choice answers or class labels.
- `BLEU-1/4, METEOR, Rouge, BERT-F1` — range: [0, 1]
  - Standard NLP sequence matching and embedding-based metrics for evaluating generated clinical reports.
- `LLM-as-a-judge score` — range: other
  - Scored by GPT-5 on diagnostic accuracy, analysis completeness, and relevance based on predefined criteria (e.g., 30-point scale).

## Input / output format

**Input**: Raw ECG waveforms (1-lead or multi-lead, sampled at 256 Hz) tokenized into 1-second segments, interleaved with text prompts, questions, or instruction templates.

**Output**: Predicted class labels or probabilities, generated clinical reports, multiple-choice answers, or numerical statistical values (e.g., RR interval, HR).

## Scoring recipe

```python
def compute_metrics(preds, gold):
    f1 = f1_score(gold, preds, average='macro')
    auroc = roc_auc_score(gold, preds_proba)
    rmse = np.sqrt(np.mean((gold - preds)**2))
    accuracy = np.mean(preds == gold)
    nlp = compute_nlp_metrics(preds, gold)
    llm = llm_as_judge(preds, gold, criteria=['accuracy', 'completeness', 'relevance'])
    return {'F1': f1, 'AUROC': auroc, 'RMSE': rmse, 'Accuracy': accuracy, 'NLP': nlp, 'LLM_Judge': llm}
```

## Common pitfalls

- Forecasting F1 drops as the prediction horizon increases, but improves with longer input windows (up to 600s).
- LLM-based baselines struggle on datasets with large label spaces (e.g., HEEDB, CSN) compared to non-LLM architectures.
- Linear probing evaluation freezes the model and trains a linear classifier on only 1% of the training data.
- LLM-as-a-judge scoring relies on GPT-5 to evaluate diagnostic accuracy, completeness, and relevance, which may introduce model-specific biases.

## Evidence (verbatim from paper)

> We report F1 score for zero-shot evaluation and AUROC for linear probing results.

## Citation

```bibtex
@misc{velingker2026camel,
  title={CAMEL: An ECG Language Model for Forecasting Cardiac Events},
  author={Velingker et al. (2026)},
  year={2026},
  note={arXiv:2602.15677}
}
```

- arXiv: 2602.15677