Chexbench Eval

---
name: chexbench-eval
description: Evaluates vision-language foundation models on chest X-ray interpretation across two axes: image perception (view classification, disease identification/classification, VQA, reasoning) and textual understanding (findings generation, summarization). It probes clinical reasoning, visual grounding, and medical text generation capabilities. Use when the user wants to benchmark on MIMIC-CXR, CheXpert, SIIM, RSNA, OpenI, SLAKE, Rad-Restruct, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.12208
  bibtex_key: chen2024chexagent
  confidence: high
---

# chexbench-eval

> A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation — Chen et al. (2024) (arXiv:2401.12208, 2024)

## What this evaluates

Evaluates vision-language foundation models on chest X-ray interpretation across two axes: image perception (view classification, disease identification/classification, VQA, reasoning) and textual understanding (findings generation, summarization). It probes clinical reasoning, visual grounding, and medical text generation capabilities.

## Datasets

- **MIMIC-CXR** — total ?; splits: test (-1)
- **CheXpert** — total ?; splits: test (-1)
- **SIIM** — total ?; splits: test (-1)
- **RSNA** — total ?; splits: test (-1)
- **OpenI** — total ?; splits: test (-1)
- **SLAKE** — total ?; splits: test (-1)
- **Rad-Restruct** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted options in multiple-choice tasks. Computed as correct predictions divided by total predictions.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence-based recall/precision/f1 score for text generation tasks.
- `CheXbert-Score` — range: [0, 1]
  - Automated metric measuring factual correctness of generated radiology findings against reference reports using a clinical NLP model.
- `RadGraph-Score` — range: [0, 1]
  - Automated metric evaluating clinical entity and relation extraction accuracy in generated reports.

## Input / output format

**Input**: A chest X-ray image paired with a text prompt (multiple-choice question with options, or an instruction to generate/summarize findings).

**Output**: For perception tasks: the selected multiple-choice option. For generation tasks: free-text radiology findings or a concise summary statement.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(predictions)

def compute_generation_metrics(predictions, references):
    # Uses standard implementations for ROUGE-L, CheXbert-Score, RadGraph-Score
    rouge_l = rouge_l_score(predictions, references)
    chexbert = chexbert_score(predictions, references)
    radgraph = radgraph_score(predictions, references)
    return rouge_l, chexbert, radgraph
```

## Common pitfalls

- Data leakage: Several baseline models (e.g., XrayGPT, RadFM) were fine-tuned on datasets included in the benchmark (OpenI, SLAKE), potentially inflating their reported accuracy.
- Log-likelihood selection: The benchmark avoids open-ended generation evaluation for perception tasks by scoring log-likelihoods of multiple-choice options, which may not capture true clinical reasoning or hallucination rates.
- Hard negatives: The image-text reasoning task uses carefully crafted hard negatives that are exceptionally difficult for all evaluated models, making performance gaps small and potentially masking model limitations.

## Evidence (verbatim from paper)

> In line with prior benchmarks designed for general domain FMs, we use a multiple-choice format, where an image and a question are posed to the FM and multiple options are considered. Since open-ended, free-text outputs from FMs are challenging to evaluate, we instead compute log-likelihood scores associated with each option; the option with the highest score is then selected as the response. For each task, we report accuracy.

## Citation

```bibtex
@misc{chen2024chexagent,
  title={A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2401.12208}
}
```

- arXiv: 2401.12208