Rexrank Eval

---
name: rexrank-eval
description: Evaluates AI models' ability to generate accurate and clinically relevant radiology reports from chest X-ray images. It assesses both linguistic quality and clinical entity extraction/alignment across diverse clinical datasets. Use when the user wants to benchmark on ReXGradient, MIMIC-CXR, IU X-ray, CheXpert Plus, or asks about evaluating this task. Reports 1/RadCliQ-v1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.15122
  bibtex_key: [REDACTED]
  confidence: high
---

# rexrank-eval

> ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation — Zhang et al. (2024) (arXiv:2411.15122, 2024)

## What this evaluates

Evaluates AI models' ability to generate accurate and clinically relevant radiology reports from chest X-ray images. It assesses both linguistic quality and clinical entity extraction/alignment across diverse clinical datasets.

## Datasets

- **ReXGradient** — total 10000; splits: test (-1)
- **MIMIC-CXR** — total ?; splits: test (-1)
- **IU X-ray** — total ?; splits: test (-1)
- **CheXpert Plus** — total ?; splits: test (-1)

## Metrics

- `1/RadCliQ-v1` **(primary)** — range: other
  - Inverse of the RadCliQ-v1 clinical entity alignment score. Higher values indicate better clinical accuracy. Computed as 1 / RadCliQ_v1_score.
- `BLEU-2` — range: [0, 1]
  - Standard 2-gram BLEU score measuring n-gram overlap between generated and reference reports.
- `BertScore` — range: [0, 1]
  - F1 score computed using contextual embeddings from a BERT model to measure semantic similarity.
- `SembScore` — range: [0, 1]
  - Semantic similarity score based on sentence embeddings, measuring contextual alignment.
- `RadGraph` — range: [0, 1]
  - F1 score measuring the overlap of extracted radiological entities and relations between prediction and ground truth.
- `RaTEScore` — range: [0, 1]
  - Radiology Text Embedding score measuring semantic similarity using domain-specific embeddings.
- `GREEN` — range: [0, 1]
  - Generative Radiology Evaluation metric assessing clinical correctness and completeness of generated reports.
- `1/FineRadScore` — range: other
  - Inverse of the FineRadScore, which evaluates fine-grained clinical correctness. Higher values indicate better performance.

## Input / output format

**Input**: Chest X-ray image(s) (and optionally clinical metadata).

**Output**: Radiology report text, generated either as 'Findings' only or 'Findings + Impression'.

## Scoring recipe

```python
def compute_metrics(pred_report, gold_report):
    radcliq = compute_radcliq_v1(pred_report, gold_report)
    bleu2 = compute_bleu2(pred_report, gold_report)
    bertscore = compute_bertscore(pred_report, gold_report)
    sembscore = compute_sembscore(pred_report, gold_report)
    radgraph = compute_radgraph(pred_report, gold_report)
    rtescore = compute_rtescore(pred_report, gold_report)
    green = compute_green(pred_report, gold_report)
    fineradscore = compute_fineradscore(pred_report, gold_report)
    return {
        '1/RadCliQ-v1': 1.0 / radcliq,
        'BLEU-2': bleu2,
        'BertScore': bertscore,
        'SembScore': sembscore,
        'RadGraph': radgraph,
        'RaTEScore': rtescore,
        'GREEN': green,
        '1/FineRadScore': 1.0 / fineradscore
    }
```

## Common pitfalls

- The primary metric is inverted (1/RadCliQ-v1), so higher values indicate better performance, contrary to standard RadCliQ scores.
- Models are evaluated on two distinct tasks: 'Findings' only and 'Findings + Impression', which should not be mixed when comparing results.
- Confidence intervals (95% CI) are reported alongside mean scores, indicating statistical variance that should be considered when ranking models.

## Evidence (verbatim from paper)

> Table 1, 2, 3 and 4 summarize the performance of various medical report generation models across four different datasets: ReXGradient, MIMIC-CXR, IU X-ray and CheXpert Plus. Among these, MedVersa demonstrates superior performance, achieving the best 1/RadCliQ-v1 scores on ReXGradient (1.01 ± 0.01), MIMIC-CXR (1.10 ± 0.02), and IU X-ray (1.46 ± 0.03) on the findings section.

## Citation

```bibtex
@misc{zhang2024rexrank,
  title={ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2411.15122}
}
```

- arXiv: 2411.15122