VisA Mvtec Ad Eval

---
name: visA-mvtec-ad-eval
description: Evaluates industrial anomaly detection and segmentation capabilities by measuring how well self-supervised pre-training methods transfer to identifying surface defects. It probes a model's ability to localize fine-grained anomalies in highly imbalanced, high-resolution industrial imagery under both one-class and few-shot supervised regimes. Use when the user wants to benchmark on VisA, MVTec-AD, or asks about evaluating this task. Reports AU-PR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.14315
  bibtex_key: zou2022spot
  confidence: high
---

# visA-mvtec-ad-eval

> SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation — Yang Zou et al. (2022) (arXiv:2207.14315, 2022)

## What this evaluates

Evaluates industrial anomaly detection and segmentation capabilities by measuring how well self-supervised pre-training methods transfer to identifying surface defects. It probes a model's ability to localize fine-grained anomalies in highly imbalanced, high-resolution industrial imagery under both one-class and few-shot supervised regimes.

## Datasets

- **VisA** — total 10821; splits: test (-1)
- **MVTec-AD** — total 5354; splits: test (-1)

## Metrics

- `AU-PR` **(primary)** — range: [0, 1]
  - Area Under the Precision-Recall curve. Computed by integrating precision over recall thresholds for predicted anomaly scores against ground truth labels or masks.
- `AU-ROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve. Computed by integrating the True Positive Rate over the False Positive Rate thresholds for predicted anomaly scores.

## Input / output format

**Input**: High-resolution industrial images (normal and anomalous) used for pre-training and downstream anomaly detection/segmentation tasks.

**Output**: Per-image anomaly score (classification) and per-pixel binary anomaly mask (segmentation).

## Scoring recipe

```python
def compute_aupr(y_true, y_pred):
    precisions, recalls, _ = precision_recall_curve(y_true, y_pred)
    return auc(recalls, precisions)

def compute_auroc(y_true, y_pred):
    fprs, tprs, _ = roc_curve(y_true, y_pred)
    return auc(fprs, tprs)
```

## Common pitfalls

- Relying on AU-ROC as the sole metric, which saturates near 1.0 and masks poor performance on highly imbalanced industrial anomaly data.
- Confusing the 1-class (unsupervised/one-class) and 2-class (supervised) evaluation regimes, which exhibit significantly different performance gaps and baseline rankings.
- Overlooking the distinction between high-shot (full anomaly set) and low-shot (5/10 anomaly samples) training conditions, which drastically changes model behavior and metric trends.

## Evidence (verbatim from paper)

> While we report both AU-PR and AU-ROC, the former metric is more relevant to the application and we see that self-supervised methods are improved up to AU-PR of $2.6\%$. ... Moreover, the gap between low AU-PR and high AU-ROC for both VisA/MVTec segmentation justifies the inflated performance view of AU-ROC, in favor of AU-PR as a more suitable metric in imbalanced datasets.

## Citation

```bibtex
@misc{zou2022spot,
  title={SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation},
  author={Yang Zou et al. (2022)},
  year={2022},
  note={arXiv:2207.14315}
}
```

- arXiv: 2207.14315