Chexficient Eval

---
name: chexficient-eval
description: Evaluates a chest X-ray vision-language foundation model across zero-shot classification, cross-modal retrieval, and adapted downstream tasks (classification, segmentation, report generation). It specifically probes data and compute efficiency, as well as the model's ability to represent long-tailed thoracic diseases without aggressive scaling. Use when the user wants to benchmark on SIIM-PTX, Pneumonia2017, TBX11K, CheXpert, MIMIC-CXR, ChestX-ray14, VinDr-CXR, VinDr-PCXR, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.22843
  bibtex_key: wang2026chexficient
  confidence: high
---

# chexficient-eval

> A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling — Chong Wang et al. (2026) (arXiv:2602.22843, 2026)

## What this evaluates

Evaluates a chest X-ray vision-language foundation model across zero-shot classification, cross-modal retrieval, and adapted downstream tasks (classification, segmentation, report generation). It specifically probes data and compute efficiency, as well as the model's ability to represent long-tailed thoracic diseases without aggressive scaling.

## Datasets

- **SIIM-PTX** — total 1372; splits: test (1372)
- **Pneumonia2017** — total 624; splits: test (624)
- **TBX11K** — total 1800; splits: test (1800)
- **CheXpert** — total 500; splits: test (500)
- **MIMIC-CXR** — total 3082; splits: test (3082)
- **ChestX-ray14** — total 25596; splits: test (25596)
- **VinDr-CXR** — total 3000; splits: test (3000)
- **VinDr-PCXR** — total 1397; splits: test (1397)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve; measures the probability that a randomly chosen positive instance is ranked higher than a negative instance. Reported as macro-average for multi-label tasks.
- `Recall@1` — range: [0, 1]
  - Fraction of queries where the correct paired image or report is retrieved within the top-1 result.
- `Dice score` — range: [0, 1]
  - 2 * |A ∩ B| / (|A| + |B|), measuring overlap between predicted and ground-truth segmentation masks.
- `RadGraph` — range: percent
  - Standard radiology report generation metric that evaluates clinical concept matching between generated and reference reports.

## Input / output format

**Input**: Chest X-ray images paired with radiology reports. For zero-shot tasks: image + class description prompt (classification) or image/report as query (retrieval). For downstream tasks: image for classification/segmentation; image + reference report for generation.

**Output**: Classification labels (binary or multi-label), retrieved report/image, segmentation mask, or generated radiology report.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    # AUROC (macro-average for multi-label)
    auroc_scores = [roc_auc_score(gold[:, i], preds[:, i]) for i in range(num_classes)]
    auroc = np.mean(auroc_scores)
    # Recall@1
    recall_at_1 = np.mean([1 if p == g else 0 for p, g in zip(preds, gold)])
    # Dice
    dice = 2 * np.sum(pred_mask * gold_mask) / (np.sum(pred_mask) + np.sum(gold_mask))
    return {'AUROC': auroc, 'Recall@1': recall_at_1, 'Dice': dice}
```

## Common pitfalls

- Confusing zero-shot evaluation (no weight updates) with adapted downstream fine-tuning (linear probing or head fine-tuning).
- Reporting single-label AUROC instead of macro-averaged AUROC for multi-disease benchmarks like CheXpert or VinDr-CXR.
- Combining Findings and Impressions sections for cross-modal retrieval without reporting section-specific retrieval performance.

## Evidence (verbatim from paper)

> The area under the receiver operating curve (AUROC), Recall@1, Dice score, and standard radiology report generation metrics (e.g., RadGraph) are utilized for evaluating the task performance of these models.

## Citation

```bibtex
@misc{wang2026chexficient,
  title={A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling},
  author={Chong Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.22843}
}
```

- arXiv: 2602.22843