Chest Xray Report Generation Eval

---
name: chest-xray-report-generation-eval
description: This evaluation probes a model's ability to generate clinically coherent and accurate natural language reports from chest X-ray images. It measures both surface-level linguistic similarity to ground-truth radiology reports and the clinical correctness of extracted pathological findings. Use when the user wants to benchmark on Indiana U. Chest X-Ray, MIMIC-CXR, or asks about evaluating this task. Reports BLEU-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2107.07314
  bibtex_key: najdenkoska2021variational
  confidence: high
---

# chest-xray-report-generation-eval

> Variational Topic Inference for Chest X-Ray Report Generation — Najdenkoska et al. (2021) (arXiv:2107.07314, 2021)

## What this evaluates

This evaluation probes a model's ability to generate clinically coherent and accurate natural language reports from chest X-ray images. It measures both surface-level linguistic similarity to ground-truth radiology reports and the clinical correctness of extracted pathological findings.

## Datasets

- **Indiana U. Chest X-Ray** — total 3195; splits: train (-1), val (-1), test (-1)
- **MIMIC-CXR** — total 218101; splits: train (-1), val (-1), test (-1)

## Metrics

- `BLEU-1` **(primary)** — range: [0, 1]
  - Computes unigram precision between generated and reference reports, penalizing overly short outputs via a brevity penalty. Scores range from 0 to 1.
- `ROUGE` — range: [0, 1]
  - Measures recall-based n-gram overlap (typically ROUGE-L) between generated and reference reports. Scores range from 0 to 1.
- `METEOR` — range: [0, 1]
  - Combines unigram precision and recall with penalties for synonymy, stemming, and word order mismatches. Scores range from 0 to 1.
- `Clinical F1` — range: [0, 1]
  - Harmonic mean of precision and recall computed on binary pathological labels extracted from reports using the CheXpert rule-based labeler. Scores range from 0 to 1.

## Input / output format

**Input**: Chest X-ray radiograph images normalized and resized to 224×224 pixels, processed through a pre-trained DenseNet-121 to extract visual features.

**Output**: A natural language chest X-ray report consisting of concatenated, lower-cased, and tokenized findings and impressions.

## Scoring recipe

```python
def evaluate(predictions, references):
    # NLG metrics
    bleu_1 = nltk.translate.bleu_score.sentence_bleu([ref.split() for ref in references], pred.split(), weights=(1,0,0,0))
    rouge = rouge_scorer.score(ref, pred)['rougeL'].fmeasure
    meteor = meteor_score.score(ref, pred)
    # Clinical metrics via CheXpert labeler
    pred_labels = chexpert_labeler(pred)
    gold_labels = chexpert_labeler(ref)
    tp = sum(p & g for p, g in zip(pred_labels, gold_labels))
    fp = sum(p & ~g for p, g in zip(pred_labels, gold_labels))
    fn = sum(~p & g for p, g in zip(pred_labels, gold_labels))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return bleu_1, rouge, meteor, precision, recall, f1
```

## Common pitfalls

- Standard NLG metrics (BLEU/ROUGE) inherently penalize paraphrasing and diversity, which contradicts the model's goal of capturing radiological uncertainty and sentence-level variability.
- Clinical efficacy scores depend entirely on the CheXpert rule-based labeler, which may misclassify or miss subtle findings compared to expert radiologists.
- Generated reports are systematically longer than ground truth, which can artificially inflate recall-based metrics like ROUGE and F1.

## Evidence (verbatim from paper)

> We adopt commonly used evaluation metrics for natural language generation (NLG), including BLEU, METEOR and ROUGE. We compare to several other neural network based state-of-the-art methods: [[17], [19], [32], [33], [10], [3]] for Indiana U. X-Rays, and [[19], [3], [20]] for MIMIC-CXR. As shown in Table 1, our VTI achieves comparable performance or yields higher scores in terms of BLEU-1-2-3, ROUGE (for Indiana U. Chest X-ray) and METEOR (for MIMIC-CXR). As an additional evaluation in terms of the clinical coherence and correctness, we employ clinical efficacy metrics, i.e., precision, recall and F1 score [[19]] to compare the extracted labels by the rule-based CheXpert labeler [[9]] for the ground-truth and generated reports.

## Citation

```bibtex
@misc{najdenkoska2021variational,
  title={Variational Topic Inference for Chest X-Ray Report Generation},
  author={Najdenkoska et al. (2021)},
  year={2021},
  note={arXiv:2107.07314}
}
```

- arXiv: 2107.07314