Cxr Abnormality Localization Eval

---
name: cxr-abnormality-localization-eval
description: Evaluates the capability of object detection models to localize thoracic abnormalities in chest X-rays under a weakly semi-supervised setting. It specifically probes how well models can leverage sparse point-level annotations alongside a small fraction of fully bounding-box-labeled images to achieve accurate region detection. Use when the user wants to benchmark on RSNA, VinDr-CXR, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.01988
  bibtex_key: ji2022benchmark
  confidence: high
---

# cxr-abnormality-localization-eval

> A Benchmark for Weakly Semi-Supervised Abnormality Localization in Chest X-Rays — Haoqin Ji et al. (2022) (arXiv:2209.01988, 2022)

## What this evaluates

Evaluates the capability of object detection models to localize thoracic abnormalities in chest X-rays under a weakly semi-supervised setting. It specifically probes how well models can leverage sparse point-level annotations alongside a small fraction of fully bounding-box-labeled images to achieve accurate region detection.

## Datasets

- **RSNA** — total 26684; splits: train (-1), test (-1)
- **VinDr-CXR** — total 15000; splits: train (-1), test (-1)

## Metrics

- `mAP` **(primary)** — range: percent
  - Mean Average Precision across all abnormality classes, computed by averaging the Area Under the Precision-Recall curve for each class at a fixed IoU threshold.

## Input / output format

**Input**: Chest X-ray images paired with either point-level annotations (for the majority of training data) or full bounding box annotations (for a randomly sampled subset of training data at 5%, 10%, 20%, 30%, 40%, or 50% ratios). Test set images are evaluated against ground-truth bounding boxes.

**Output**: Bounding box predictions (coordinates and confidence scores) for each detected abnormality class.

## Scoring recipe

```python
def compute_mAP(predictions, ground_truth, iou_threshold=0.5):
    ap_scores = []
    for class_id in all_classes:
        preds = [p for p in predictions if p['class'] == class_id]
        gts = [g for g in ground_truth if g['class'] == class_id]
        preds.sort(key=lambda x: x['score'], reverse=True)
        tp, fp = [], []
        matched_gts = set()
        for pred in preds:
            best_iou = 0
            best_idx = -1
            for i, gt in enumerate(gts):
                if i not in matched_gts:
                    iou = calculate_iou(pred['box'], gt['box'])
                    if iou > best_iou:
                        best_iou, best_idx = iou, i
            if best_iou >= iou_threshold:
                tp.append(1); fp.append(0); matched_gts.add(best_idx)
            else:
                tp.append(0); fp.append(1)
        precisions = np.cumsum(tp) / (np.cumsum(tp) + np.cumsum(fp) + 1e-8)
        recalls = np.cumsum(tp) / (len(gts) + 1e-8)
        ap_scores.append(np.trapz(precisions, recalls))
    return sum(ap_scores) / len(ap_scores) * 100
```

## Common pitfalls

- Unclear lesion boundaries cause region proposals from image-level weakly supervised baselines to be highly inaccurate (mAP ≤ 5%).
- VinDr-CXR exhibits a long-tailed class distribution; categories with fewer than ten samples must be grouped into an 'Others' class to stabilize training.
- Point-level annotations are highly sensitive to exact lesion center placement, requiring consistency regularization to prevent performance degradation.

## Evidence (verbatim from paper)

> The mean average precision (mAP) is adopted as the evaluation metric. Note that we also evaluate several image-level-annotation-based weakly supervised approaches on the two datasets. However, due to the unclear boundaries of lesion areas, the region proposals are totally inaccurate, which results in an mAP ≤ 5%. Hence, we do not include the results in the benchmark.

## Citation

```bibtex
@misc{ji2022benchmark,
  title={A Benchmark for Weakly Semi-Supervised Abnormality Localization in Chest X-Rays},
  author={Haoqin Ji et al. (2022)},
  year={2022},
  note={arXiv:2209.01988}
}
```

- arXiv: 2209.01988