Imagenet O Eval

---
name: imagenet-o-eval
description: Evaluates out-of-distribution (OOD) detection capabilities by measuring how well models assign low confidence to images of objects that do not exist in their training distribution. It probes whether models can reliably distinguish in-distribution classes from novel anomalies without relying on spurious cues. Use when the user wants to benchmark on ImageNet-O, or asks about evaluating this task. Reports AUPR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1907.07174
  bibtex_key: hendrycks2019naturaladversarialexamples
  confidence: high
---

# imagenet-o-eval

> Natural Adversarial Examples — Hendrycks et al. (2019) (arXiv:1907.07174, 2019)

## What this evaluates

Evaluates out-of-distribution (OOD) detection capabilities by measuring how well models assign low confidence to images of objects that do not exist in their training distribution. It probes whether models can reliably distinguish in-distribution classes from novel anomalies without relying on spurious cues.

## Datasets

- **ImageNet-O** — total ?; splits: test (-1); repo https://github.com/hendrycks/natural-adv-examples

## Metrics

- `AUPR` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve computed using anomaly scores defined as the negative of the maximum softmax probability across the 200 ImageNet-O classes.

## Input / output format

**Input**: RGB images of out-of-distribution anomalies (e.g., triceratops, T-Rex) not present in the model's training classes.

**Output**: Maximum softmax probability across the 200 ImageNet-O classes (used to derive anomaly scores).

## Scoring recipe

```python
anomaly_scores = [-max(softmax_probs) for probs in model_outputs]
precision, recall, _ = precision_recall_curve(gold_labels, anomaly_scores)
aupr = auc(recall, precision)
```

## Common pitfalls

- Random chance AUPR is ~16.67%, so scores must be interpreted relative to this baseline rather than assuming higher is always linearly better.
- ImageNet-O images may overlap with ImageNet-21K training data, which can artificially inflate OOD detection performance if not accounted for.

## Evidence (verbatim from paper)

> Our metric for assessing out-of-distribution detection performance of ImageNet-O examples is the area under the precision-recall curve (AUPR). This metric requires anomaly scores. Our anomaly score is the negative of the maximum softmax probabilities from a model that can classify the 200 ImageNet-O classes.

## Citation

```bibtex
@misc{hendrycks2019naturaladversarialexamples,
  title={Natural Adversarial Examples},
  author={Hendrycks et al. (2019)},
  year={2019},
  note={arXiv:1907.07174}
}
```

- arXiv: 1907.07174