Ecg Grounding Eval

---
name: ecg-grounding-eval
description: Evaluates a multimodal LLM's ability to perform reliable, evidence-based ECG interpretation under full and missing modality conditions. It probes diagnostic accuracy, clinical reasoning fidelity, cross-modal consistency, and real-world clinical utility compared to cardiologist standards. Use when the user wants to benchmark on ECG-Grounding test set, or asks about evaluating this task. Reports Diagnosis Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.04279
  bibtex_key: jin2026ecgr1
  confidence: high
---

# ecg-grounding-eval

> ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation — Jin et al. (2026) (arXiv:2602.04279, 2026)

## What this evaluates

Evaluates a multimodal LLM's ability to perform reliable, evidence-based ECG interpretation under full and missing modality conditions. It probes diagnostic accuracy, clinical reasoning fidelity, cross-modal consistency, and real-world clinical utility compared to cardiologist standards.

## Datasets

- **ECG-Grounding test set** — total 2381; splits: test (2381)

## Metrics

- `Diagnosis Accuracy` **(primary)** — range: [0, 10] or [0, 100]
  - Rubric-based score quantifying diagnostic correctness. Graded by DeepSeek-V3.1-Terminus using predefined clinical criteria.
- `Analysis Completeness` — range: [0, 10] or [0, 100]
  - Rubric-based score measuring coverage of critical diagnostic elements and ECG components in the interpretation.
- `Analysis Relevance` — range: [0, 10] or [0, 100]
  - Rubric-based score assessing whether generated findings are clinically aligned with the final diagnosis.
- `Lead Evidence Validity` — range: [0, 10] or [0, 100]
  - Rubric-based score evaluating lead-specific evidence relevance rather than template-style enumeration.
- `ECG Feature Grounding` — range: [0, 10] or [0, 100]
  - Rubric-based score measuring grounding in verifiable ECG features.
- `Evidence-Based Reasoning` — range: [0, 10] or [0, 100]
  - Rubric-based score quantifying the fidelity of stepwise clinical reasoning.
- `Clinical Diagnostic Fidelity` — range: [0, 10] or [0, 100]
  - Rubric-based score assessing the tightness of evidence-to-diagnosis linkage.
- `BLEU-4` — range: [0, 1]
  - Text-similarity metric computing n-gram precision (up to 4-grams) between time-series-only and image-only model outputs.
- `ROUGE-L` — range: [0, 1]
  - Text-similarity metric measuring longest common subsequence recall/precision between modality-specific outputs.
- `SBERT-Score` — range: [0, 1]
  - Semantic similarity score using sentence embeddings to quantify cross-modal output agreement.

## Input / output format

**Input**: ECG time-series signals and/or ECG images (interleaved modalities), optionally with clinical prompts or protocol-guided instructions.

**Output**: Structured clinical interpretation containing final diagnosis, fine-grained evidence localization, lead-wise findings, and stepwise reasoning (rhythm, conduction, morphology, ischemia assessment).

## Scoring recipe

```python
def score_rubric(model_output, gold_diagnosis, rubric):
    prompt = f'Rubric: {rubric}\nGold: {gold_diagnosis}\nModel: {model_output}'
    score = llm_grader(prompt)  # DeepSeek-V3.1-Terminus returns float per dimension
    return score

def score_consistency(ts_output, img_output):
    bleu = compute_bleu_4(ts_output, img_output)
    rouge = compute_rouge_l(ts_output, img_output)
    sbert = compute_sbert_similarity(ts_output, img_output)
    return {'BLEU-4': bleu, 'ROUGE-L': rouge, 'SBERT-Score': sbert}

def score_human_eval(model_outputs, cardiologists):
    ratings = [c.score(output) for c in cardiologists for output in model_outputs]
    return mean(ratings), std(ratings)
```

## Common pitfalls

- Non-specialized MLLMs often achieve high Analysis Completeness but low Analysis Relevance and Lead Evidence Validity due to systematic hallucinations, misleading aggregate scores.
- Cross-modal consistency relies on text-similarity metrics (BLEU/ROUGE/SBERT) which may not capture clinical nuance or factual alignment between modality-specific outputs.
- Automated grading with LLMs (e.g., DeepSeek-V3.1-Terminus) can drift from human cardiologist judgments; direct human validation is recommended for clinical deployment claims.

## Evidence (verbatim from paper)

> Among all non-ECG-specialized MLLMs, GPT-5.1 achieves the highest Diagnosis Accuracy. However, its absolute score remains low at 31.48, falling far short of the reliability required for real-world clinical deployment.

## Citation

```bibtex
@misc{jin2026ecgr1,
  title={ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation},
  author={Jin et al. (2026)},
  year={2026},
  note={arXiv:2602.04279}
}
```

- arXiv: 2602.04279