Iu Xray Report Gen Eval

---
name: iu-xray-report-gen-eval
description: Evaluates a vision-language model's ability to generate clinically accurate and semantically coherent radiology reports from chest X-ray images. It probes the model's capacity for medical terminology usage, anatomical consistency, and structured clinical text generation. Use when the user wants to benchmark on IU X-ray, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.03178
  bibtex_key: jahangir2025llamaxr
  confidence: high
---

# iu-xray-report-gen-eval

> LLaMA-XR: A Novel Framework for Radiology Report Generation using LLaMA and QLoRA Fine Tuning — Md. Zihad Bin Jahangir et al. (2025) (arXiv:2506.03178, 2025)

## What this evaluates

Evaluates a vision-language model's ability to generate clinically accurate and semantically coherent radiology reports from chest X-ray images. It probes the model's capacity for medical terminology usage, anatomical consistency, and structured clinical text generation.

## Datasets

- **IU X-ray** — total ?; splits: test (-1)

## Metrics

- `ROUGE-L` **(primary)** — range: [0, 1]
  - Measures the longest common subsequence (LCS) between generated and reference reports to assess structural and semantic fidelity.
- `METEOR` — range: [0, 1]
  - Computes alignment between generated and reference text using synonymy, stemming, and semantic matching to evaluate linguistic fluency and semantic accuracy.

## Input / output format

**Input**: Chest X-ray image (optionally with attention map visualization)

**Output**: Natural language radiology report describing findings, impressions, and anatomical observations.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    rouge_l_scores = [rouge_scorer.score(ref, pred)['rougeL'].fmeasure for pred, ref in zip(predictions, references)]
    meteor_scores = [meteor_score([ref], pred) for pred, ref in zip(predictions, references)]
    return {
        'ROUGE-L': sum(rouge_l_scores) / len(predictions),
        'METEOR': sum(meteor_scores) / len(predictions)
    }
```

## Common pitfalls

- Table 3 results for baseline methods are quoted from their original publications, not re-evaluated on the same split.
- BLEU metrics heavily penalize synonymy and paraphrasing, making them less reliable for clinical text than ROUGE-L or METEOR.
- The dataset split sizes and exact preprocessing steps for IU X-ray are not detailed in this paper.

## Evidence (verbatim from paper)

> LLaMA-XR demonstrates superior performance on ROUGE-L and METEOR, two widely used metrics that emphasize semantic fidelity and syntactic fluency. In particular, LLaMA-XR achieves a 4.34% improvement in ROUGE-L over the state-of-the-art best result from the SGF method.

## Citation

```bibtex
@misc{jahangir2025llamaxr,
  title={LLaMA-XR: A Novel Framework for Radiology Report Generation using LLaMA and QLoRA Fine Tuning},
  author={Md. Zihad Bin Jahangir et al. (2025)},
  year={2025},
  note={arXiv:2506.03178}
}
```

- arXiv: 2506.03178