Mvtec Eval

---
name: mvtec-eval
description: Unsupervised anomaly detection and pixel-level localization on industrial defect data. It probes the model's ability to distinguish normal from defective samples and precisely segment defect regions without using labeled anomalies during training. Use when the user wants to benchmark on MVTec, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2107.12571
  bibtex_key: gudovskiy2021cflowad
  confidence: high
---

# mvtec-eval

> CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows — Gudovskiy et al. (2021) (arXiv:2107.12571, 2021)

## What this evaluates

Unsupervised anomaly detection and pixel-level localization on industrial defect data. It probes the model's ability to distinguish normal from defective samples and precisely segment defect regions without using labeled anomalies during training.

## Datasets

- **MVTec** — total 5354; splits: train (3629), test (1725)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve. Computed for both image-level detection scores and flattened pixel-level anomaly score maps.
- `AUPRO` — range: [0, 1]
  - Area under the per-region-overlap curve. Computes the average true positive rate across multiple overlap thresholds between predicted and ground truth masks, ensuring equal importance for large and small anomalies.

## Input / output format

**Input**: RGB images resized to class-specific resolutions (128x128, 256x256, or 512x512) without cropping. Training images undergo ±5° rotation augmentation.

**Output**: Per-image anomaly score for detection. Per-pixel anomaly score map for localization, optionally thresholded to produce binary segmentation masks.

## Scoring recipe

```python
img_scores = model.predict_image_scores(images)
auroc_det = compute_auroc(img_scores, gt_labels)

score_maps = model.predict_anomaly_maps(images)
auroc_loc = compute_auroc(score_maps.flatten(), gt_pixel_labels)
aupro_loc = compute_aupro(score_maps, gt_masks, thresholds=[0.1, 0.5])
```

## Common pitfalls

- AUROC is skewed towards large-area anomalies; AUPRO must be reported alongside to fairly evaluate small defect localization.
- Optimal input resolution varies by class (e.g., 256x256 for macro objects, 512x512 for most, 128x128 for transistor); using a fixed resolution degrades performance.
- Training is strictly unsupervised; only anomaly-free images are used. Using defect labels during training violates the protocol.

## Evidence (verbatim from paper)

> We use widely-used threshold-agnostic evaluation metrics for localization: area under the receiver operating characteristic curve (AUROC) and area under the per-region-overlap curve (AUPRO). AUROC is skewed towards large-area anomalies, while AUPRO metric ensures that both large and small anomalies are equally important in localization. Image-level AD detection is reported by the AUROC only.

## Citation

```bibtex
@misc{gudovskiy2021cflowad,
  title={CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows},
  author={Gudovskiy et al. (2021)},
  year={2021},
  note={arXiv:2107.12571}
}
```

- arXiv: 2107.12571