Im Iad Eval

---
name: im-iad-eval
description: Evaluates industrial image anomaly detection algorithms across seven manufacturing datasets under unsupervised, few-shot, continual, and fully supervised settings. It probes both image-level classification and pixel-level localization capabilities, while also measuring computational efficiency like inference speed and GPU memory. Use when the user wants to benchmark on MVTec AD, MVTec LOCO-AD, MPDD, BTAD, MTD, VisA, DAGM, or asks about evaluating this task. Reports Image AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.13359
  bibtex_key: xie2023imiad
  confidence: high
---

# im-iad-eval

> IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing — Xie et al. (2023) (arXiv:2301.13359, 2023)

## What this evaluates

Evaluates industrial image anomaly detection algorithms across seven manufacturing datasets under unsupervised, few-shot, continual, and fully supervised settings. It probes both image-level classification and pixel-level localization capabilities, while also measuring computational efficiency like inference speed and GPU memory.

## Datasets

- **MVTec AD** — total ?; splits: test (-1)
- **MVTec LOCO-AD** — total ?; splits: test (-1)
- **MPDD** — total ?; splits: test (-1)
- **BTAD** — total ?; splits: test (-1)
- **MTD** — total ?; splits: test (-1)
- **VisA** — total ?; splits: test (-1)
- **DAGM** — total ?; splits: test (-1)

## Metrics

- `Image AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic (ROC) curve for image-level binary anomaly classification. Computed by varying the classification threshold and plotting true positive rate against false positive rate.
- `Pixel AP` — range: [0, 1]
  - Average Precision for pixel-level anomaly localization. Computed from the precision-recall curve over all pixels using the anomaly heatmap scores.
- `Pixel PRO` — range: [0, 1]
  - Product-Region Overlap metric for pixel-level localization. Measures the mean overlap ratio between predicted anomaly regions and ground truth masks across multiple thresholds.

## Input / output format

**Input**: RGB images of industrial products, accompanied by binary masks for pixel-level anomalies and binary labels for image-level classification.

**Output**: Per-image anomaly score and per-pixel anomaly heatmap/mask.

## Scoring recipe

```python
def compute_auc(y_true, y_pred):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

def compute_ap(y_true, y_pred):
    precision, recall, _ = precision_recall_curve(y_true, y_pred)
    return auc(recall, precision)

def compute_pro(y_true_mask, y_pred_map, thresholds):
    overlaps = []
    for thr in thresholds:
        pred_mask = (y_pred_map > thr).astype(int)
        intersection = np.logical_and(pred_mask, y_true_mask).sum()
        union = np.logical_or(pred_mask, y_true_mask).sum()
        overlaps.append(intersection / union if union > 0 else 0)
    return np.mean(overlaps)
```

## Common pitfalls

- Image-level and pixel-level metrics often diverge; a model can excel at one but fail at the other, so reporting only one is misleading.
- Logical anomalies (displacement/missing parts) are structurally different from dents/scratches and require global feature extraction that many structural detectors lack.
- Efficiency metrics (inference time, GPU memory) are critical for real-world deployment but are frequently secondary to accuracy in academic benchmarks.

## Evidence (verbatim from paper)

> Researchers commonly use image-level and pixel-level metrics to evaluate the classification performance of IAD algorithms. In practice, the image-level metric is used to judge whether the whole product is abnormal, while the pixel-level metric indicates anomaly localization performance... According to Table VI, specific IAD methods, like PatchCore, perform well on image AUROC but poorly on pixel AP, or vice versa.

## Citation

```bibtex
@misc{xie2023imiad,
  title={IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing},
  author={Xie et al. (2023)},
  year={2023},
  note={arXiv:2301.13359}
}
```

- arXiv: 2301.13359