Pad Eval

---
name: pad-eval
description: Evaluates the ability of anomaly detection models to identify and localize defects in 3D objects from unseen camera poses without requiring pose alignment. It probes pose-invariant representation learning and robustness to viewpoint changes in both pixel-level segmentation and image-level classification. Use when the user wants to benchmark on MAD, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.07716
  bibtex_key: zhou2023pad
  confidence: high
---

# pad-eval

> PAD: A Dataset and Benchmark for Pose-agnostic Anomaly Detection — Zhou et al. (2023) (arXiv:2310.07716, 2023)

## What this evaluates

Evaluates the ability of anomaly detection models to identify and localize defects in 3D objects from unseen camera poses without requiring pose alignment. It probes pose-invariant representation learning and robustness to viewpoint changes in both pixel-level segmentation and image-level classification.

## Datasets

- **MAD** — total 11000; splits: train (-1), test (-1); repo https://github.com/EricLee0224/PAD

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, calculated as the integral of the true positive rate (R_TP) with respect to the false positive rate (R_FP): AUROC = ∫(R_TP) dR_FP. Computed separately for pixel-level segmentation and image-level classification.

## Input / output format

**Input**: RGB images of objects captured from multiple viewpoints (poses), containing either normal samples or samples with anomalies.

**Output**: Pixel-level anomaly segmentation masks and image-level anomaly classification scores/labels.

## Scoring recipe

```python
def compute_auroc(predictions, ground_truth):
    # predictions: flattened array of anomaly scores (pixel-level or image-level)
    # ground_truth: flattened array of binary labels (0 for normal, 1 for anomaly)
    fpr, tpr, _ = roc_curve(ground_truth, predictions)
    return auc(fpr, tpr)
```

## Common pitfalls

- Reconstruction-based methods may fail to generate accurate normal references for unseen poses, leading to a disconnect between high AUROC scores and poor visual reconstruction quality.
- Image-level detection performance often degrades significantly in the pose-agnostic setting compared to pixel-level localization, as holistic normal features are harder to learn from sparse viewpoints.

## Evidence (verbatim from paper)

> Following previous work, we specifically choose the Area Under the Receiver Operating Caracteristic Curve (AUROC) as the primary metric for evaluating the performance of anomaly segmentation at the pixel-level and anomaly classification at the image-level. While there exist various evaluation metric for these tasks, AUROC stands out as the most widely used and suitable metric for conducting comprehensive benchmarking. The AUROC score can be calculated as follows: AUROC = ∫(R_TP) dR_FP

## Citation

```bibtex
@misc{zhou2023pad,
  title={PAD: A Dataset and Benchmark for Pose-agnostic Anomaly Detection},
  author={Zhou et al. (2023)},
  year={2023},
  note={arXiv:2310.07716}
}
```

- arXiv: 2310.07716