Chestx Det Eval

---
name: chestx-det-eval
description: Evaluates instance-level detection and segmentation of thoracic diseases on chest X-rays. It probes a model's ability to localize and classify 13 disease categories while handling domain-specific challenges like ambiguous boundaries and disease co-occurrence. Use when the user wants to benchmark on ChestX-Det, DR-private, or asks about evaluating this task. Reports AP_50^bb.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.10326
  bibtex_key: lian2021sar-net
  confidence: high
---

# chestx-det-eval

> A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation — Lian et al. (2021) (arXiv:2104.10326, 2021)

## What this evaluates

Evaluates instance-level detection and segmentation of thoracic diseases on chest X-rays. It probes a model's ability to localize and classify 13 disease categories while handling domain-specific challenges like ambiguous boundaries and disease co-occurrence.

## Datasets

- **ChestX-Det** — total 3575; splits: train (3025), val (302), test (553); repo https://github.com/Deepwise-AILab/ChestX-Det-Dataset
- **DR-private** — total 6629; splits: train (5800), val (580), test (829)

## Metrics

- `AP_50^bb` **(primary)** — range: percent
  - Average Precision for bounding boxes at an Intersection-over-Union (IoU) threshold of 0.5. Computed by matching predicted boxes to ground truth based on confidence scores and IoU ≥ 0.5, then calculating the area under the precision-recall curve.
- `AP_25^bb` — range: percent
  - Bounding box Average Precision at IoU threshold 0.25. Used as a reference metric to better reflect classification performance.
- `AP_75^bb` — range: percent
  - Bounding box Average Precision at IoU threshold 0.75. Used as a reference metric to better reflect localization performance.
- `AP_mask` — range: percent
  - Average Precision for instance segmentation masks at IoU threshold 0.5.
- `AP_50^mask` — range: percent
  - Mask Average Precision at IoU threshold 0.5.
- `AP_75^mask` — range: percent
  - Mask Average Precision at IoU threshold 0.75.
- `Recall@0.1fp/image` — range: percent
  - Instance-level recall computed at a fixed rate of 0.1 false positives per image. Provides a direct clinical reference for sensitivity at low false alarm rates.

## Input / output format

**Input**: Front-view chest X-ray images. Ground truth includes bounding boxes and instance-level masks for 13 disease categories plus a normal class.

**Output**: Predicted bounding boxes and instance segmentation masks with associated class labels and confidence scores.

## Scoring recipe

```python
def compute_ap(predictions, ground_truth, iou_thresh=0.5):
    matches = []
    for gt in ground_truth:
        best_pred = max(predictions, key=lambda p: iou(p.box, gt.box))
        if iou(best_pred.box, gt.box) >= iou_thresh and not best_pred.used:
            matches.append(True)
            best_pred.used = True
        else:
            matches.append(False)
    predictions.sort(key=lambda p: p.confidence, reverse=True)
    tp = [m for m in matches if m]
    fp = [not m for m in matches if m] + [True] * (len(predictions) - len(matches))
    # Standard COCO-style precision-recall curve and AP calculation
    return ap_score(tp, fp)

def compute_recall_at_fp(predictions, ground_truth, fp_per_img=0.1, num_imgs=1):
    predictions.sort(key=lambda p: p.confidence, reverse=True)
    tp, fp = 0, 0
    for pred in predictions:
        if iou(pred.box, ground_truth[0].box) >= 0.5: tp += 1
        else: fp += 1
        if fp >= fp_per_img * num_imgs: return tp / len(ground_truth)
    return tp / len(ground_truth)
```

## Common pitfalls

- Disease regions often lack clear boundaries, making high-IoU metrics (AP_75) unreliable and harder to optimize than classification metrics.
- Overlapping diseases with similar spatial distributions (e.g., fibrosis and emphysema) frequently cause classification confusion, leading to swapped predictions.
- The dataset is imbalanced across disease categories, so mean AP may mask poor performance on rare abnormalities.

## Evidence (verbatim from paper)

> We borrow the metric of bounding box AP (AP^bb) [5] used in general object detection. Considering that disease/abnormality regions lack clear boundary as general objects, we adopt AP_50^bb as the main evaluation metric. AP_25^bb and AP_75^bb are used for reference. We believe that AP_75^bb better reflects localization performance and AP_25^bb better reflects classification performance. AP_50^bb is the comprehensive performance index for classification and localization. For more practical usage, we also present instance-level recall at fixed FP(false positive) per image for more direct reference. In addition, we use mask AP (AP^mask) to evaluate the performance of instance segmentation.

## Citation

```bibtex
@misc{lian2021sar-net,
  title={A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation},
  author={Lian et al. (2021)},
  year={2021},
  note={arXiv:2104.10326}
}
```

- arXiv: 2104.10326