Ecg Expert Qa Eval

---
name: ecg-expert-qa-eval
description: Evaluates medical large language models on heart disease diagnosis using expert-validated QA pairs. It probes clinical reasoning, risk-aware decision-making, and patient-centric interaction capabilities across multiple diagnostic sub-tasks. Use when the user wants to benchmark on ECG-Expert-QA, or asks about evaluating this task. Reports BLEU-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.17475
  bibtex_key: wang2025ecgexpertqa
  confidence: high
---

# ecg-expert-qa-eval

> ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis — Wang et al. (2025) (arXiv:2502.17475, 2025)

## What this evaluates

Evaluates medical large language models on heart disease diagnosis using expert-validated QA pairs. It probes clinical reasoning, risk-aware decision-making, and patient-centric interaction capabilities across multiple diagnostic sub-tasks.

## Datasets

- **ECG-Expert-QA** — total 47211; splits: test (-1); repo https://github.com/Zaozzz/ECG-Expert-QA

## Metrics

- `BLEU-1` **(primary)** — range: [0, 1]
  - BP * p1, where p1 is modified unigram precision and BP is brevity penalty (1 if candidate length > reference length, else exp(1 - reference length / candidate length)).
- `ROUGE-L` — range: [0, 1]
  - F_beta = ((1+beta^2)*Precision*Recall)/(Recall+beta^2*Precision), where Precision = LCS/len(candidate), Recall = LCS/len(reference), and beta is typically 1.
- `METEOR` — range: [0, 1]
  - F_mean * (1 - Penalty), where F_mean = 10*P*R/(9P+R) and Penalty = 0.5*(chunks/matches)^3, incorporating synonym matching and stemming.
- `Model-to-Model Scoring (MMS)` — range: other
  - Automated scoring by a reference LLM (Model A) evaluating a target model (Model B) on semantic consistency, content coverage, and language fluency against ground truth.

## Input / output format

**Input**: Clinical context or question related to ECG/diagnosis, formatted as a QA prompt.

**Output**: Natural language answer/response generated by the LLM.

## Scoring recipe

```python
def evaluate(candidates, references):
    bleu_scores = [compute_bleu1(c, r) for c, r in zip(candidates, references)]
    rouge_scores = [compute_rouge_l(c, r) for c, r in zip(candidates, references)]
    meteor_scores = [compute_meteor(c, r) for c, r in zip(candidates, references)]
    mms_scores = [llm_judge_score(c, r) for c, r in zip(candidates, references)]
    return {
        'BLEU-1': sum(bleu_scores) / len(bleu_scores),
        'ROUGE-L': sum(rouge_scores) / len(rouge_scores),
        'METEOR': sum(meteor_scores) / len(meteor_scores),
        'MMS': sum(mms_scores) / len(mms_scores)
    }
```

## Common pitfalls

- BLEU-1 only measures unigram lexical overlap and fails to capture semantic meaning or clinical accuracy.
- Model-to-Model Scoring (MMS) relies on a single LLM as judge, which may introduce systematic bias or hallucination in clinical contexts.
- Sub-datasets vary significantly in context length, causing token-limit failures for lightweight models like MiniMind2 if not carefully filtered.

## Evidence (verbatim from paper)

> To evaluate the performance of LLMs on the ECG-Expert-QA dataset, we adopted four widely used evaluation metrics in natural language generation (NLG): BLEU-1*[[24]]*, ROUGE-L*[[25]]*, and METEOR*[[26]]*. These metrics measure the similarity between the model-generated answers and ground-truth references in terms of lexical overlap and semantic relevance.

## Citation

```bibtex
@misc{wang2025ecgexpertqa,
  title={ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.17475}
}
```

- arXiv: 2502.17475