Chestxray8 Eval

---
name: chestxray8-eval
description: Evaluates weakly-supervised multi-label classification and spatial localization of eight common thoracic diseases on chest X-rays. It probes a model's ability to detect disease presence from image-level labels and localize pathological regions using only bounding box annotations during testing. Use when the user wants to benchmark on ChestX-ray8, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1705.02315
  bibtex_key: wang2017chestxray8
  confidence: high
---

# chestxray8-eval

> ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases — Wang et al. (2017) (arXiv:1705.02315, 2017)

## What this evaluates

Evaluates weakly-supervised multi-label classification and spatial localization of eight common thoracic diseases on chest X-rays. It probes a model's ability to detect disease presence from image-level labels and localize pathological regions using only bounding box annotations during testing.

## Datasets

- **ChestX-ray8** — total 108948; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve computed per disease class across the test set.
- `Localization Accuracy (Acc.)` — range: [0, 1]
  - Proportion of correctly localized instances where Intersection over Bounding Box area (IoBB) exceeds a threshold T(IoBB).
- `Average False Positives (AFP)` — range: other
  - Mean number of false positive bounding box detections per image for each disease class.

## Input / output format

**Input**: Frontal-view chest X-ray images (typically 1024x1024 pixels).

**Output**: Multi-label classification probabilities for 8 disease classes and predicted bounding boxes for disease localization.

## Scoring recipe

```python
def score_classification(y_true, y_pred):
    auc_scores = {}
    for disease in diseases:
        auc_scores[disease] = roc_auc_score(y_true[disease], y_pred[disease])
    return auc_scores

def score_localization(pred_boxes, gt_boxes, threshold=0.1):
    correct = 0
    total_gt = len(gt_boxes)
    false_pos = 0
    for gt in gt_boxes:
        if any(iou_box_area(pred, gt) > threshold for pred in pred_boxes):
            correct += 1
        else:
            false_pos += 1
    acc = correct / total_gt if total_gt > 0 else 0
    return acc, false_pos
```

## Common pitfalls

- Training uses only image-level labels; bounding boxes are strictly for test-time evaluation, not supervised training.
- Localization metric uses IoBB (Intersection over predicted box area) rather than standard IoU, making thresholds like 0.1 or 0.25 much more lenient than typical object detection benchmarks.
- Significant class imbalance exists (e.g., Pneumonia <1% of dataset), heavily skewing per-class AUC and localization accuracy.

## Evidence (verbatim from paper)

> The corresponding Area-Under-Curve (AUC) values are given in Table 3.

## Citation

```bibtex
@misc{wang2017chestxray8,
  title={ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases},
  author={Wang et al. (2017)},
  year={2017},
  note={arXiv:1705.02315}
}
```

- arXiv: 1705.02315