Sim3d Eval

---
name: sim3d-eval
description: Evaluates 3D anomaly detection and segmentation capabilities in industrial settings using multiview and multimodal (image + depth) inputs. It probes a model's ability to identify and localize defects across multiple object categories under both in-domain (real-to-real) and out-of-domain (synthetic-to-real) conditions. Use when the user wants to benchmark on SiM3D, or asks about evaluating this task. Reports I-AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.02328
  bibtex_key: costanzino2026modulateandmap
  confidence: high
---

# sim3d-eval

> Modulate-and-Map: Crossmodal Feature Mapping with Cross-View Modulation for 3D Anomaly Detection — Costanzino et al. (2026) (arXiv:2604.02328, 2026)

## What this evaluates

Evaluates 3D anomaly detection and segmentation capabilities in industrial settings using multiview and multimodal (image + depth) inputs. It probes a model's ability to identify and localize defects across multiple object categories under both in-domain (real-to-real) and out-of-domain (synthetic-to-real) conditions.

## Datasets

- **SiM3D** — total ?; splits: test (-1)

## Metrics

- `I-AUROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve for image-level anomaly detection. Computed per object category and averaged across all categories.
- `V-AUPRO@1%` — range: [0, 1]
  - Area under the Precision-Recall curve at a fixed 1% false positive rate for voxel-level anomaly segmentation. Computed per category and averaged.

## Input / output format

**Input**: Multiview image and depth/point cloud data for industrial objects. Features are extracted per view and projected into a unified 3D space for anomaly scoring.

**Output**: 3D anomaly volumes containing per-voxel anomaly scores and per-category binary segmentation masks, plus a global anomaly score per object.

## Scoring recipe

```python
def compute_metrics(gold_labels, pred_scores, gold_masks, pred_masks):
    # Detection: AUROC per category, then mean
    i_auroc_per_cat = [compute_auroc(gold_labels[c], pred_scores[c]) for c in categories]
    mean_i_auroc = np.mean(i_auroc_per_cat)
    
    # Segmentation: AUPRO at 1% FPR per category, then mean
    v_aupro_per_cat = [compute_aupro(gold_masks[c], pred_masks[c], fpr=0.01) for c in categories]
    mean_v_aupro = np.mean(v_aupro_per_cat)
    
    return mean_i_auroc, mean_v_aupro
```

## Common pitfalls

- Synthetic-to-real domain shift causes severe performance drops and erratic behavior in some baselines, making cross-domain robustness a key differentiator.
- Baselines must be adapted from 2D to 3D by processing each view independently and aggregating via projection, which can introduce alignment errors if not handled correctly.
- Missing table entries indicate some competitors were not originally evaluated on SiM3D, so direct comparison requires careful adaptation of their original 2D pipelines.

## Evidence (verbatim from paper)

> We evaluate our method against state-of-the-art anomaly detection approaches adapted to the multiview and multimodal 3D anomaly detection scenario set forth by SiM3D [12]. The considered competitors include memory bank methods (PatchCore [27], BTF [19], M3DM [36]), teacher-student approaches (EfficientAD [1], AST [29]), and the original, single-view Crossmodal Feature Mapping (CFM [11]). Hence, as proposed in [12], all the competitors are adapted to produce 3D anomaly volumes by processing each view independently and aggregating the resulting 2D anomaly maps into the 3D space using the projection strategy described in the SiM3D paper. Tab. 1 and Tab. 2 report results on the real-to-real and synthetic-to-real setups of SiM3D, respectively, with qualitative results shown in Fig. 5. Real-to-real Setup. In the real-to-real setup (Tab. 1), our method (MODMAP) achieves the best performance in both the detection and segmentation tasks, with a mean I-AUROC of 0.844 and a mean V-AUPRO@ 1% of 0.804.

## Citation

```bibtex
@misc{costanzino2026modulateandmap,
  title={Modulate-and-Map: Crossmodal Feature Mapping with Cross-View Modulation for 3D Anomaly Detection},
  author={Costanzino et al. (2026)},
  year={2026},
  note={arXiv:2604.02328}
}
```

- arXiv: 2604.02328