Mmad Bbox Eval

---
name: mmad-bbox-eval
description: Evaluates the fine-grained localization capability of vision-language models on industrial anomaly detection. It measures how accurately models can predict bounding boxes around defects compared to ground-truth annotations and human experts. Use when the user wants to benchmark on MMAD-BBox, or asks about evaluating this task. Reports BBox-Mask IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.13779
  bibtex_key: jiang2026adcopilot
  confidence: high
---

# mmad-bbox-eval

> AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison — Jiang et al. (2026) (arXiv:2603.13779, 2026)

## What this evaluates

Evaluates the fine-grained localization capability of vision-language models on industrial anomaly detection. It measures how accurately models can predict bounding boxes around defects compared to ground-truth annotations and human experts.

## Datasets

- **MMAD-BBox** — total ?; splits: test (-1)

## Metrics

- `BBox-Mask IoU` **(primary)** — range: [0, 1]
  - Converts predicted and ground-truth bounding boxes into binary masks and computes Intersection over Union (IoU) in the mask space. Additionally reports classification accuracy at IoU thresholds of 0.5, 0.3, 0.2, and 0.1.

## Input / output format

**Input**: Industrial images paired with evaluation prompts requiring anomaly localization.

**Output**: Text-formatted bounding box coordinates predicting the anomaly region.

## Scoring recipe

```python
def compute_bbox_mask_iou(pred_boxes, gt_boxes):
    pred_masks = boxes_to_binary_masks(pred_boxes)
    gt_masks = boxes_to_binary_masks(gt_boxes)
    intersection = np.logical_and(pred_masks, gt_masks).sum(axis=(1,2))
    union = np.logical_or(pred_masks, gt_masks).sum(axis=(1,2))
    iou_scores = intersection / np.maximum(union, 1e-6)
    mean_iou = iou_scores.mean()
    accuracy_at_thresholds = {t: (iou_scores >= t).mean() for t in [0.5, 0.3, 0.2, 0.1]}
    return mean_iou, accuracy_at_thresholds
```

## Common pitfalls

- Using conventional box-matching IoU instead of mask-space IoU, which fails for irregular or fragmented industrial defects.
- Applying standard object detection IoU thresholds (e.g., 0.5) without accounting for the inherent difficulty of anomaly localization, which requires lower thresholds (0.1-0.3) for meaningful evaluation.

## Evidence (verbatim from paper)

> Conventional box-matching IoU metrics can be unreliable, as industrial defects often have irregular shapes, disconnected fragments, or ambiguous instance boundaries. To provide a fairer and more consistent evaluation, we propose a new metric called BBox-Mask IoU. As illustrated in Fig.[4](#S5.F4 "Figure 4 ‣ V-B Implementation Details ‣ V Model Training ‣ AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison"), we convert both the predicted boxes and the ground-truth boxes into binary masks and compute IoU in the mask space. This alleviates bias caused by differing output granularities and offers a smooth, fine-grained measure of localization accuracy. In addition to the mean IoU, we report accuracy under IoU thresholds of 0.5, 0.3, 0.2, and 0.1, where relatively low thresholds are adopted as anomaly localization is inherently more challenging than standard object detection.

## Citation

```bibtex
@misc{jiang2026adcopilot,
  title={AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison},
  author={Jiang et al. (2026)},
  year={2026},
  note={arXiv:2603.13779}
}
```

- arXiv: 2603.13779