Hopkins Rfo Bench Eval

---
name: hopkins-rfo-bench-eval
description: Evaluates object detection models on chest X-rays for identifying critical retained foreign objects (RFOs) like sponges and needles. It probes both classification accuracy and precise localization of rare medical anomalies under data-scarce conditions. Use when the user wants to benchmark on Hopkins RFOs Bench, or asks about evaluating this task. Reports ACC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.06937
  bibtex_key: wang2025rfo_bench
  confidence: high
---

# hopkins-rfo-bench-eval

> Dataset and Benchmark for Enhancing Critical Retained Foreign Object Detection — Wang et al. (2025) (arXiv:2507.06937, 2025)

## What this evaluates

Evaluates object detection models on chest X-rays for identifying critical retained foreign objects (RFOs) like sponges and needles. It probes both classification accuracy and precise localization of rare medical anomalies under data-scarce conditions.

## Datasets

- **Hopkins RFOs Bench** — total 144; splits: train (-1), test (-1); repo https://github.com/YuliWanghust/RFO_Bench

## Metrics

- `ACC` **(primary)** — range: [0, 1]
  - Accuracy = (True Positives + True Negatives) / Total Instances. Measures the proportion of correctly classified images at a fixed confidence threshold.
- `FNR` — range: [0, 1]
  - False Negative Rate = False Negatives / (False Negatives + True Positives). Measures the proportion of actual RFO cases missed by the model.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Quantifies the model's ability to discriminate between positive and negative classes across all classification thresholds.
- `FROC` — range: [0, 1]
  - Free-response Receiver Operating Characteristic. Plots the true positive fraction against the average number of false positives per image across varying detection thresholds to evaluate localization performance.

## Input / output format

**Input**: Chest X-ray radiograph images.

**Output**: Bounding box coordinates and class labels (presence of critical RFO) for each detected object.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, iou_thresh=0.5):
    tp, fp, fn = 0, 0, 0
    for pred in predictions:
        if match_to_gold(pred, ground_truth, iou=iou_thresh): tp += 1
        else: fp += 1
    fn = len(ground_truth) - tp
    acc = (tp + tn) / len(ground_truth)  # tn derived from image-level negatives
    fnr = fn / (fn + tp)
    auc = compute_roc_auc(tp, fp)
    froc = compute_froc_curve(tp, fp, num_images=len(ground_truth))
    return acc, fnr, auc, froc
```

## Common pitfalls

- Using excessive synthetic augmentation (e.g., 4,000 images) can degrade performance due to overfitting or diminishing returns, with 2,000 images often yielding optimal results.
- DDPM-based synthetic data frequently reduces detection performance compared to physics-based generation, highlighting fidelity and generalizability limitations for rare clinical features.
- Evaluation is strictly confined to the held-out Hopkins RFOs Bench testing set, so results do not generalize to external hospital datasets or different RFO types.

## Evidence (verbatim from paper)

> For classification, we use ACC, FNR, and AUC metrics. Localization performance is evaluated using the FROC metric.

## Citation

```bibtex
@misc{wang2025rfo_bench,
  title={Dataset and Benchmark for Enhancing Critical Retained Foreign Object Detection},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2507.06937}
}
```

- arXiv: 2507.06937