Mvtec Ad Eval

---
name: mvtec-ad-eval
description: Evaluates an image classification and segmentation model's ability to detect and localize defects in industrial products without seeing anomalous examples during training. It probes the model's capacity to learn nominal feature distributions and identify deviations at both image and pixel levels. Use when the user wants to benchmark on MVTec AD, Magnetic Tile Defects (MTD), Mini Shanghai Tech Campus (mSTC), or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.08265
  bibtex_key: roth2021totalrecall
  confidence: high
---

# mvtec-ad-eval

> Towards Total Recall in Industrial Anomaly Detection — Roth et al. (2021) (arXiv:2106.08265, 2021)

## What this evaluates

Evaluates an image classification and segmentation model's ability to detect and localize defects in industrial products without seeing anomalous examples during training. It probes the model's capacity to learn nominal feature distributions and identify deviations at both image and pixel levels.

## Datasets

- **MVTec AD** — total 5354; splits: train (-1), test (1725)
- **Magnetic Tile Defects (MTD)** — total 1317; splits: train (-1), test (-1)
- **Mini Shanghai Tech Campus (mSTC)** — total ?; splits: train (-1), test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve computed over image-level anomaly scores. On MVTec AD, the class-average AUROC is reported.
- `Pixel-wise AUROC` — range: [0, 1]
  - Area under the ROC curve computed over pixel-level anomaly scores for segmentation masks.
- `PRO` — range: [0, 1]
  - Measures the overlap and recovery of connected anomaly components to better account for varying anomaly sizes in industrial datasets.
- `Error` — range: percent
  - Sum of false positives and false negatives at the F1-optimal threshold.

## Input / output format

**Input**: RGB images resized and center-cropped to 256×256 or 224×224. No data augmentation is applied. Training data contains only nominal (defect-free) images.

**Output**: Per-image anomaly score for classification. Per-pixel anomaly map/mask for segmentation.

## Scoring recipe

```python
def evaluate(scores, labels, masks=None, gt_masks=None):
    class_auroc = np.mean([roc_auc_score(labels[c], scores[c]) for c in np.unique(labels)])
    best_f1, best_err = 0, float('inf')
    for t in np.unique(scores):
        preds = (scores >= t).astype(int)
        fp, fn = np.sum((preds==1)&(labels==0)), np.sum((preds==0)&(labels==1))
        err = fp + fn
        f1 = 2 * (1 - err/len(labels)) if len(labels) > 0 else 0
        if f1 > best_f1: best_f1, best_err = f1, err
    error_pct = (best_err / len(labels)) * 100
    pw_auroc = roc_auc_score(gt_masks.flatten(), masks.flatten())
    pro = compute_pro(gt_masks, masks)
    return class_auroc, pw_auroc, pro, error_pct
```

## Common pitfalls

- MVTec AD requires class-average AUROC, not image-average.
- PRO metric evaluates connected component overlap/recovery, not simple pixel-wise IoU.
- Error rate is computed at the F1-optimal threshold, which must be searched per dataset/method.
- Training uses only nominal data; anomalous images are strictly for testing.

## Evidence (verbatim from paper)

> Image-level anomaly detection performance is measured via the area under the receiver-operator curve (AUROC) using produced anomaly scores. In accordance with prior work we compute on MVTec the class-average AUROC. To measure segmentation performance, we use both pixel-wise AUROC and the PRO metric first, both following[6]. The PRO score takes into account the overlap and recovery of connected anomaly components to better account for varying anomaly sizes in MVTec AD, see[6] for details.

## Citation

```bibtex
@misc{roth2021totalrecall,
  title={Towards Total Recall in Industrial Anomaly Detection},
  author={Roth et al. (2021)},
  year={2021},
  note={arXiv:2106.08265}
}
```

- arXiv: 2106.08265