Rad Robustness Eval

---
name: rad-robustness-eval
description: Evaluates the robustness of image anomaly detection models against real-world imaging distortions, including free viewpoints, uneven illumination, and motion blur. It measures how well unsupervised and zero-shot methods localize and classify anomalies on industrial work platforms with foreign objects. Use when the user wants to benchmark on RAD, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.07176
  bibtex_key: cheng2024rad
  confidence: high
---

# rad-robustness-eval

> RAD: A Comprehensive Dataset for Benchmarking the Robustness of Image Anomaly Detection — Cheng et al. (2024) (arXiv:2406.07176, 2024)

## What this evaluates

Evaluates the robustness of image anomaly detection models against real-world imaging distortions, including free viewpoints, uneven illumination, and motion blur. It measures how well unsupervised and zero-shot methods localize and classify anomalies on industrial work platforms with foreign objects.

## Datasets

- **RAD** — total ?; splits: test (-1); repo https://github.com/hustCYQ/RAD-dataset

## Metrics

- `AUROC` **(primary)** — range: percent
  - Area Under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
- `Max-F1` — range: percent
  - Maximal F1 score. The highest F1 score achieved across all possible classification thresholds on the predicted anomaly scores.
- `AP` — range: percent
  - Average Precision. The area under the Precision-Recall curve, summarizing model performance across all thresholds.
- `AUPRO` — range: percent
  - Area Under Per Region Overlap curve. Evaluates pixel-level localization by computing the overlap ratio between predicted and ground truth anomaly masks for each image, then averaging across the dataset.

## Input / output format

**Input**: Resized 224×224 RGB images of industrial work platforms containing foreign objects (bolts, cable ties, sponges, tapes) under varying conditions (normal, uneven illumination, blurry, free views).

**Output**: Image-level anomaly score/prediction and pixel-wise anomaly score map or binary mask.

## Scoring recipe

```python
# Image-level
auroc = roc_auc_score(y_true_img, y_score_img)
ap = average_precision_score(y_true_img, y_score_img)
max_f1 = max(f1_score(y_true_img, (y_score_img > t)) for t in np.linspace(0, 1, 100))
# Pixel-level
aupro = compute_aupro(y_true_masks, y_score_maps) # AUPRO averages per-image overlap ratios
```

## Common pitfalls

- AUPRO is exclusively for pixel-level evaluation and should not be reported for image-level results.
- All input images are resized to 224×224, which may degrade pixel-level localization precision compared to original high-resolution captures.
- Zero-shot methods (WinCLIP, APRIL-GAN, SAA) are evaluated without any training on the RAD dataset, relying entirely on pre-trained foundation models.

## Evidence (verbatim from paper)

> Following the mainstream setting as in MVTec-AD, we employ several key metrics to evaluate the performance of anomaly detection methods, namely the Area Under the Receiver Operating Characteristic curve (AUROC), Maximal F1 score (Max-F1), Average Precision (AP), and Area Under Per Region Overlap (AUPRO). The first three indicators (AUROC, Max-F1, AP) are utilized for both image-level and pixel-level anomaly detection evaluation, and AUPRO is exclusively employed for pixel-level anomaly detection evaluation.

## Citation

```bibtex
@misc{cheng2024rad,
  title={RAD: A Comprehensive Dataset for Benchmarking the Robustness of Image Anomaly Detection},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2406.07176}
}
```

- arXiv: 2406.07176