Ood Detection Eval

---
name: ood-detection-eval
description: Evaluates a model's ability to distinguish between in-distribution (ID) samples and out-of-distribution (OOD) samples. It measures how well the model's decision boundaries separate known classes from unknown data distributions using various synthetic and natural OOD benchmarks. Use when the user wants to benchmark on SVHN, CIFAR-10, CIFAR-100, TinyImageNet, TinyImageNet-crop, TinyImageNet-resize, LSUN-crop, LSUN-resize, iSUN, or asks about evaluating this task. Reports TNR@TPR95.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.13916
  bibtex_key: [REDACTED]
  confidence: high
---

# ood-detection-eval

> A Novel Data Augmentation Technique for Out-of-Distribution Sample Detection using Compounded Corruptions — Hebbalaguppe et al. (2022) (arXiv:2207.13916, 2022)

## What this evaluates

Evaluates a model's ability to distinguish between in-distribution (ID) samples and out-of-distribution (OOD) samples. It measures how well the model's decision boundaries separate known classes from unknown data distributions using various synthetic and natural OOD benchmarks.

## Datasets

- **SVHN** — total ?; splits: train (-1), test (-1)
- **CIFAR-10** — total ?; splits: train (-1), test (-1)
- **CIFAR-100** — total ?; splits: train (-1), test (-1)
- **TinyImageNet** — total ?; splits: train (-1), test (-1)
- **TinyImageNet-crop** — total ?; splits: test (-1)
- **TinyImageNet-resize** — total ?; splits: test (-1)
- **LSUN-crop** — total ?; splits: test (-1)
- **LSUN-resize** — total ?; splits: test (-1)
- **iSUN** — total ?; splits: test (-1)

## Metrics

- `TNR@TPR95` **(primary)** — range: percent
  - True Negative Rate at a fixed True Positive Rate of 95%. Computed from the ROC curve of OOD scores on ID vs OOD samples.
- `AUROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, measuring the trade-off between true positive and false positive rates across all thresholds.
- `Detection Error` — range: percent
  - Classification error rate for OOD detection, typically calculated as the average of false positive and false negative rates at the optimal threshold, or 1 - AUROC.
- `ID Acc` — range: percent
  - Standard top-1 classification accuracy on the in-distribution test set.

## Input / output format

**Input**: RGB images (32x32 or 64x64 pixels) fed into a trained neural network classifier to extract logits or softmax probabilities.

**Output**: Per-class probability scores or logits. OOD scores are derived (e.g., 1 - max softmax probability), and metrics are computed per dataset and averaged across all OOD benchmarks.

## Scoring recipe

```python
# 1. Compute OOD score for each sample (e.g., 1 - max_softmax(logits))
# 2. Split scores into ID (label=0) and OOD (label=1) arrays
# 3. Compute ROC curve: fpr, tpr, _ = roc_curve(y_true, ood_scores)
# 4. TNR@TPR95: Find FPR where TPR >= 0.95, convert to percentage
# 5. AUROC: Integrate ROC curve (trapz)
# 6. DetErr: (FPR + FNR) / 2 at operating point, or 1 - AUROC
# 7. Average each metric across all OOD datasets
# 8. Report mean ± std over 3 independent runs
```

## Common pitfalls

- Main table reports metrics averaged across all OOD benchmark datasets, not per-dataset values.
- CnC results include standard deviation over 3 runs, while baseline methods often report single-point values.
- OOD datasets include both natural images (LSUN, iSUN) and processed TinyImageNet variants, requiring consistent preprocessing pipelines.

## Evidence (verbatim from paper)

> Evaluation Metrics: We compare the performance of various approaches using TNR@TPR95, AUROC and Detection Error. See Suppl. for description on evaluation metrics. Values are averaged over all OOD benchmark datasets. CnC reasults are averaged on 3 evaluation runs.

## Citation

```bibtex
@misc{hebbalaguppe2022cnc,
  title={A Novel Data Augmentation Technique for Out-of-Distribution Sample Detection using Compounded Corruptions},
  author={Hebbalaguppe et al. (2022)},
  year={2022},
  note={arXiv:2207.13916}
}
```

- arXiv: 2207.13916