Xray Report Gen Eval

---
name: xray-report-gen-eval
description: Evaluates the capability of vision-language models to generate accurate and clinically relevant radiology reports from chest X-ray images. It probes both linguistic fluency and coverage against reference reports, as well as clinical accuracy in identifying pathological findings. Use when the user wants to benchmark on IU X-ray, MIMIC-CXR, CheXpert Plus, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.16776
  bibtex_key: zhang2025emrrg
  confidence: high
---

# xray-report-gen-eval

> EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation — Zhang et al. (2025) (arXiv:2510.16776, 2025)

## What this evaluates

Evaluates the capability of vision-language models to generate accurate and clinically relevant radiology reports from chest X-ray images. It probes both linguistic fluency and coverage against reference reports, as well as clinical accuracy in identifying pathological findings.

## Datasets

- **IU X-ray** — total 7470; splits: train (-1), test (-1), val (-1)
- **MIMIC-CXR** — total 227835; splits: train (270790), val (2130), test (3858)
- **CheXpert Plus** — total 187711; splits: train (40463), val (5780), test (11562)

## Metrics

- `BLEU-1` — range: [0, 1]
  - 1-gram overlap between generated and reference reports, averaged over the test set.
- `BLEU-2` — range: [0, 1]
  - 2-gram overlap between generated and reference reports, averaged over the test set.
- `BLEU-3` — range: [0, 1]
  - 3-gram overlap between generated and reference reports, averaged over the test set.
- `BLEU-4` — range: [0, 1]
  - 4-gram overlap between generated and reference reports, averaged over the test set.
- `ROUGE-L` **(primary)** — range: [0, 1]
  - Longest common subsequence recall and precision between generated and reference reports, averaged over the test set.
- `METEOR` — range: [0, 1]
  - Harmonic mean of precision and recall with synonym/stem matching, averaged over the test set.
- `CIDEr` — range: [0, 1]
  - TF-IDF weighted n-gram similarity between generated and reference reports, averaged over the test set.
- `Precision` — range: [0, 1]
  - Clinical metric: TP/(TP+FP), where TP/FP are correctly/incorrectly identified pathological findings.
- `Recall` — range: [0, 1]
  - Clinical metric: TP/(TP+FN), where TP/FN are correctly identified/missed pathological findings.
- `F1` — range: [0, 1]
  - Clinical metric: 2×P×R/(P+R), harmonic mean of Precision and Recall for pathological findings.

## Input / output format

**Input**: Chest X-ray images (frontal and/or lateral views) paired with patient demographic/contextual metadata.

**Output**: Free-text radiology report, typically structured into sections such as Findings, Impression, Indication, and Comparison.

## Scoring recipe

```python
def score(predictions, references):
    nlg = {'BLEU-4': 0, 'ROUGE-L': 0, 'METEOR': 0, 'CIDEr': 0}
    tp, fp, fn = 0, 0, 0
    for pred, ref in zip(predictions, references):
        for m in ['BLEU-4', 'ROUGE-L', 'METEOR', 'CIDEr']:
            nlg[m] += compute_metric(m, pred, ref)
        pred_set, ref_set = extract_diseases(pred), extract_diseases(ref)
        tp += len(pred_set & ref_set)
        fp += len(pred_set - ref_set)
        fn += len(ref_set - pred_set)
    n = len(predictions)
    nlg = {k: v/n for k, v in nlg.items()}
    p = tp/(tp+fp) if (tp+fp)>0 else 0
    r = tp/(tp+fn) if (tp+fn)>0 else 0
    f1 = 2*p*r/(p+r) if (p+r)>0 else 0
    return {**nlg, 'Precision': p, 'Recall': r, 'F1': f1}
```

## Common pitfalls

- Ground truth selection varies across baselines; this protocol uses the 'Findings' section, whereas some competitors use 'Impression' or concatenated sections, making direct score comparison invalid without alignment.
- Dataset splits are not standardized; this work follows R2GenGPT/R2GenCSR partition protocols rather than official dataset splits, requiring exact match of train/val/test indices for fair comparison.
- Clinical metrics (Precision/Recall/F1) depend on the specific disease annotation scheme (e.g., RadGraph vs. CheXpert labels), which is not explicitly detailed in the metric definition section.

## Evidence (verbatim from paper)

> For evaluation metrics, we adopt natural language metrics and clinical metrics to evaluate our generated X-ray reports. For the natural language metrics, we choose BLEU, ROUGE-L, METEOR, and CIDEr. For the clinical metrics, i.e., Precision, Recall, and F1-measure, the formula can be formally defined as: Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2×P×R/(P+R) where TP (True Positive) refers to instances that are correctly identified as positive, FP (False Positive) denotes cases incorrectly labeled as positive when they are actually negative (also known as a Type I error), and FN (False Negative) represents instances incorrectly classified as negative despite being positive (referred to as a Type II error).

## Citation

```bibtex
@misc{zhang2025emrrg,
  title={EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.16776}
}
```

- arXiv: 2510.16776