Chex Eval

---
name: chex-eval
description: Evaluates a vision-language model's ability to perform interactive localization, region classification, and text generation on chest X-rays. It probes zero-shot multitask capabilities, including sentence grounding, pathology detection, and customizable report generation. Use when the user wants to benchmark on MS-CXR, VinDrCXR, NIH8, CIG, MIMIC-CXR, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.15770
  bibtex_key: muller2024chex
  confidence: high
---

# chex-eval

> ChEX: Interactive Localization and Region Description in Chest X-rays — Müller et al. (2024) (arXiv:2404.15770, 2024)

## What this evaluates

Evaluates a vision-language model's ability to perform interactive localization, region classification, and text generation on chest X-rays. It probes zero-shot multitask capabilities, including sentence grounding, pathology detection, and customizable report generation.

## Datasets

- **MS-CXR** — total ?; splits: test (-1)
- **VinDrCXR** — total ?; splits: test (-1)
- **NIH8** — total ?; splits: test (-1)
- **CIG** — total ?; splits: test (-1)
- **MIMIC-CXR** — total ?; splits: test (-1)

## Metrics

- `mAP` **(primary)** — range: percent
  - Mean Average Precision across pathology classes, computed by averaging precision-recall curves over different confidence thresholds.
- `mIoU` — range: [0, 1]
  - Mean Intersection over Union between predicted and ground-truth bounding boxes, averaged across classes.
- `AUROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve for binary region classification.
- `wAUROC` — range: [0, 1]
  - Weighted AUROC for region classification, accounting for class imbalance.
- `Mic-F1-14` — range: percent
  - Micro-averaged F1 score over 14 specific chest pathology findings.
- `Mac-F1-14` — range: percent
  - Macro-averaged F1 score over 14 specific chest pathology findings.
- `Ex-F1-14` — range: percent
  - Exact-match F1 score for predicting the presence/absence of 14 findings.
- `Mic-F1-5+` — range: percent
  - Micro-averaged F1 score over 5+ common findings.
- `Mac-F1-5+` — range: percent
  - Macro-averaged F1 score over 5+ common findings.
- `METEOR` — range: percent
  - METEOR score for evaluating generated text against reference reports, measuring alignment and synonymy.

## Input / output format

**Input**: Chest X-ray image, textual query (e.g., pathology name or regional hint), and optionally a bounding box prompt.

**Output**: Predicted bounding box coordinates, region classification label, and/or generated text description/sentence.

## Scoring recipe

```python
def evaluate(preds, gold):
    scores = {}
    scores['mAP'] = mean_average_precision(preds['boxes'], gold['boxes'])
    scores['mIoU'] = mean_iou(preds['boxes'], gold['boxes'])
    scores['AUROC'] = roc_auc_score(gold['labels'], preds['scores'])
    scores['Mic-F1-14'] = micro_f1(gold['findings'], preds['findings'])
    scores['METEOR'] = meteor_score(gold['text'], preds['text'])
    return scores
```

## Common pitfalls

- Test splits and pre-processing differ across baseline models, complicating direct numerical comparison.
- Baselines are typically specialized for single tasks, making head-to-head comparison with a multitask model inherently uneven.
- Language metrics like METEOR are reported low because the model generates aspect-level sentences rather than full reports.

## Evidence (verbatim from paper)

> On MS-CXR, ChEX improves by 25% on Mac-F1-14. On CIG, ChEX improves by 18% on Mic-F1-14, 40% on Mac-F1-14, and 29% on METEOR, although RGRG was explicitly trained on this task.

## Citation

```bibtex
@misc{muller2024chex,
  title={ChEX: Interactive Localization and Region Description in Chest X-rays},
  author={Müller et al. (2024)},
  year={2024},
  note={arXiv:2404.15770}
}
```

- arXiv: 2404.15770