Ood Pointcloud Seg Eval

---
name: ood-pointcloud-seg-eval
description: Evaluates a model's ability to detect out-of-distribution (OOD) inputs in 3D point cloud semantic segmentation. It probes domain shift robustness (indoor vs outdoor scenes) and sensor failure simulation (missing color channels) by measuring uncertainty-based OOD scores against in-distribution data. Use when the user wants to benchmark on Semantic3D, S3DIS, Semantic3D (no color), or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.06241
  bibtex_key: [REDACTED]
  confidence: high
---

# ood-pointcloud-seg-eval

> A Benchmark for Out of Distribution Detection in Point Cloud 3D Semantic Segmentation — Veeramacheneni et al. (2022) (arXiv:2211.06241, 2022)

## What this evaluates

Evaluates a model's ability to detect out-of-distribution (OOD) inputs in 3D point cloud semantic segmentation. It probes domain shift robustness (indoor vs outdoor scenes) and sensor failure simulation (missing color channels) by measuring uncertainty-based OOD scores against in-distribution data.

## Datasets

- **Semantic3D** — total ?; splits: train (-1), val (-1), test (-1)
- **S3DIS** — total ?; splits: test (-1)
- **Semantic3D (no color)** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate (in-distribution correctly classified) and false positive rate (OOD incorrectly classified as ID) across all classification thresholds.

## Input / output format

**Input**: 3D point cloud data with semantic labels (for ID) or unlabelled/unknown labels (for OOD), optionally with RGB color channels.

**Output**: Per-point semantic segmentation predictions, along with uncertainty scores (e.g., Maximum Softmax Probability or entropy) derived from Deep Ensembles or Flipout variants.

## Scoring recipe

```python
def compute_auroc(gold_labels, ood_scores):
    # gold_labels: 1 for ID, 0 for OOD
    # ood_scores: uncertainty score (higher = more OOD)
    fpr, tpr, _ = roc_curve(gold_labels, ood_scores)
    auroc = auc(fpr, tpr)
    return auroc
```

## Common pitfalls

- Confusing domain shift (Benchmark A: indoor vs outdoor) with sensor failure (Benchmark B: missing color); they require different model behaviors and uncertainty calibration.
- Using standard classification accuracy instead of uncertainty-based OOD scores (MSP/entropy) to evaluate detection capability.
- Not accounting for the fact that Deep Ensembles and Flipout/Dropout produce different uncertainty distributions, requiring consistent thresholding or ranking for AUROC.

## Evidence (verbatim from paper)

> Evaluates uncertainty-based OOD scores—Maximum Softmax Probability (MSP) and entropy—derived from Deep Ensembles and Flipout variants of RandLA-Net. Deep Ensembles achieve superior OOD detection performance (AUROC: 0.893 on Benchmark A, 0.773 on Benchmark B) due to better epistemic uncertainty estimation, outperforming Flipout and Dropout in both settings.

## Citation

```bibtex
@misc{veeramacheneni2022oodpointcloud,
  title={A Benchmark for Out of Distribution Detection in Point Cloud 3D Semantic Segmentation},
  author={Veeramacheneni et al. (2022)},
  year={2022},
  note={arXiv:2211.06241}
}
```

- arXiv: 2211.06241