Heartcare Bench Eval

---
name: heartcare-bench-eval
description: Evaluates multimodal ECG understanding and clinical reasoning across closed/open question answering, report generation, and signal prediction. It probes a model's ability to align temporal signal patterns with diagnostic language and generate clinically faithful outputs. Use when the user wants to benchmark on Heartcare-BenchS, Heartcare-BenchI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.05831
  bibtex_key: [REDACTED]
  confidence: high
---

# heartcare-bench-eval

> Heartcare Suite: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding — Xie et al. (2025) (arXiv:2506.05831, 2025)

## What this evaluates

Evaluates multimodal ECG understanding and clinical reasoning across closed/open question answering, report generation, and signal prediction. It probes a model's ability to align temporal signal patterns with diagnostic language and generate clinically faithful outputs.

## Datasets

- **Heartcare-BenchS** — total ?; splits: test (-1); repo https://github.com/DCDmllm/Heartcare-Suite
- **Heartcare-BenchI** — total ?; splits: test (-1); repo https://github.com/DCDmllm/Heartcare-Suite

## Metrics

- `accuracy` **(primary)** — range: percent
  - Computed as the percentage of correctly predicted labels for closed-ended diagnostic, waveform, and rhythm questions.
- `BERTScore-F1 (F1-Bio)` — range: [0, 1]
  - Measures semantic similarity between generated and reference answers using a BERT-based embedding space, reporting the F1 score.
- `ROUGE-L` — range: [0, 1]
  - Calculates the F1 score based on the longest common subsequence between the generated text and the reference.
- `RadGraph-F1 (F1-Rad)` — range: [0, 1]
  - Extracts clinical entities and relations from both generated and reference reports, then computes the F1 score over the matched graph structures.
- `GPT-4-based accuracy (Acc)` — range: [0, 1]
  - Uses GPT-4 to judge whether the generated report matches the reference in clinical content, outputting a binary correct/incorrect label averaged over the dataset.

## Input / output format

**Input**: Multi-lead ECG signals (provided as raw digital time-series or converted to image form) accompanied by clinical instruction prompts.

**Output**: Textual responses containing diagnostic classifications, waveform/rhythm descriptions, open-ended answers, or structured clinical reports.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type == 'closed_qa':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task_type == 'open_qa':
        return bertscore_f1(predictions, golds), rouge_l(predictions, golds)
    elif task_type == 'report_gen':
        return gpt4_accuracy(predictions, golds), radgraph_f1(predictions, golds), rouge_l(predictions, golds)
```

## Common pitfalls

- Baseline models often fail to process raw signal inputs, requiring conversion to images that lose temporal resolution.
- GPT-4-based accuracy penalizes semantically correct but differently phrased reports due to strict overlap requirements.
- Signal prediction tasks are excluded for baselines that cannot follow the instruction, potentially biasing cross-model comparisons.

## Evidence (verbatim from paper)

> Table[2] reports the results on open-ended ECG QA, evaluated using BERTScore-F1 (F1-Bio) and ROUGE-L. Tasks are divided into three subtasks: diagnosis, waveform, and rhythm.

## Citation

```bibtex
@misc{xie2025heartcare,
  title={Heartcare Suite: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2506.05831}
}
```

- arXiv: 2506.05831