Lod Ood Detection Eval

---
name: lod-ood-detection-eval
description: Evaluates a model's ability to distinguish in-distribution (ID) samples from out-of-distribution (OOD) samples using a threshold-free loss-difference clustering approach. It measures detection performance across standard benchmarks with diverse natural images and hard benchmarks where OOD classes share the same source dataset as ID. Use when the user wants to benchmark on CIFAR100, SVHN, Places, LSUN-Crop, LSUN-Resize, Textures, CIFAR10, TinyImageNet, or asks about evaluating this task. Reports FPR95.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.12952
  bibtex_key: geng2025lod
  confidence: high
---

# lod-ood-detection-eval

> LoD: Loss-difference OOD Detection by Intentionally Label-Noisifying Unlabeled Wild Data — Geng et al. (2025) (arXiv:2505.12952, 2025)

## What this evaluates

Evaluates a model's ability to distinguish in-distribution (ID) samples from out-of-distribution (OOD) samples using a threshold-free loss-difference clustering approach. It measures detection performance across standard benchmarks with diverse natural images and hard benchmarks where OOD classes share the same source dataset as ID.

## Datasets

- **CIFAR100** — total ?; splits: train (-1), test (-1)
- **SVHN** — total ?; splits: train (-1), test (-1)
- **Places** — total ?; splits: val (-1)
- **LSUN-Crop** — total ?; splits: test (-1)
- **LSUN-Resize** — total ?; splits: test (-1)
- **Textures** — total ?; splits: test (-1)
- **CIFAR10** — total ?; splits: train (-1), test (-1)
- **TinyImageNet** — total ?; splits: train (-1), val (-1)

## Metrics

- `FPR95` **(primary)** — range: percent
  - False positive rate on OOD examples when the true positive rate on ID examples is fixed at 95%.
- `AUROC` — range: percent
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all thresholds.
- `ACC` — range: percent
  - In-distribution classification accuracy, calculated as the percentage of correctly classified ID test samples.

## Input / output format

**Input**: RGB images from ID training sets, unlabeled wild data (mixed ID and OOD images), and OOD test sets. The model processes images through a backbone (WideResNet-40) to extract penultimate features for loss computation and binary OOD classification.

**Output**: Per image: a loss value (or loss difference) used for OOD scoring, and a binary OOD detection label (ID vs OOD) derived via threshold-free clustering. For ID test data, class predictions are also produced.

## Scoring recipe

```python
def compute_metrics(predictions, gold, id_preds, id_labels):
    # predictions: OOD scores (higher = more likely OOD)
    # gold: 1 for ID, 0 for OOD
    # FPR95
    tpr_target = 0.95
    threshold = np.percentile(predictions[gold==1], (1 - tpr_target) * 100)
    fpr95 = np.mean(predictions[gold==0] > threshold) * 100
    # AUROC
    auroc = roc_auc_score(gold, predictions) * 100
    # ACC
    acc = accuracy_score(id_labels, id_preds) * 100
    return fpr95, auroc, acc
```

## Common pitfalls

- The method explicitly uses a threshold-free clustering approach on loss dynamics, so applying a fixed decision threshold during evaluation contradicts the protocol.
- Hard benchmarks construct OOD data by holding out classes from the same dataset as ID (e.g., CIFAR10), which is fundamentally different from standard benchmarks that use entirely different datasets.
- The mixture proportion π controls the ratio of ID to OOD samples in the unlabeled wild training data, not the test distribution, and results vary significantly across π ∈ {0.1, 0.5, 0.9}.

## Evidence (verbatim from paper)

> we adopt the following evaluation metrics: (1) the false positive rate (FPR95) of OOD examples when true positive rate of ID examples is at 95%, (2) Area Under the Receiver Operating Characteristic curve (AUROC), and (3) ID classification Accuracy (ACC).

## Citation

```bibtex
@misc{geng2025lod,
  title={LoD: Loss-difference OOD Detection by Intentionally Label-Noisifying Unlabeled Wild Data},
  author={Geng et al. (2025)},
  year={2025},
  note={arXiv:2505.12952}
}
```

- arXiv: 2505.12952