Semi Supervised Novelty Detection Eval

---
name: semi-supervised-novelty-detection-eval
description: Evaluates a model's ability to distinguish in-distribution (ID) samples from out-of-distribution (OOD) samples in a semi-supervised setting, where only labeled ID data and unlabeled mixed data are available during training. Use when the user wants to benchmark on MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100, ImageNet, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.05825
  bibtex_key: tifrea2020semi
  confidence: high
---

# semi-supervised-novelty-detection-eval

> Semi-supervised novelty detection using ensembles with regularized disagreement — Tifrea et al. (2020) (arXiv:2012.05825, 2020)

## What this evaluates

Evaluates a model's ability to distinguish in-distribution (ID) samples from out-of-distribution (OOD) samples in a semi-supervised setting, where only labeled ID data and unlabeled mixed data are available during training.

## Datasets

- **MNIST** — total ?; splits: train (-1), test (-1)
- **FashionMNIST** — total ?; splits: train (-1), test (-1)
- **SVHN** — total ?; splits: train (-1), test (-1)
- **CIFAR10** — total ?; splits: train (-1), test (-1)
- **CIFAR100** — total ?; splits: train (-1), test (-1)
- **ImageNet** — total ?; splits: train (-1), test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
- `TNR@95` — range: [0, 1]
  - True Negative Rate measured at a fixed False Positive Rate of 5%, indicating the proportion of OOD samples correctly rejected when the ID false alarm rate is 5%.

## Input / output format

**Input**: Image or medical scan samples passed through a neural network to produce class probabilities.

**Output**: Softmax probability distribution over classes; ensemble score computed as the entropy of the averaged softmax probabilities across models.

## Scoring recipe

```python
# predictions: list of softmax outputs per ensemble member for a batch
# gold: binary labels (1 for ID, 0 for OOD)
avg_probs = np.mean(predictions, axis=0)
score = -np.sum(avg_probs * np.log(avg_probs + 1e-10), axis=1)
auroc = roc_auc_score(gold, score)
tnr95 = tnr_at_fpr(gold, score, fpr=0.05)
```

## Common pitfalls

- Pretraining on the full training set in novel-class settings leaks OOD information, artificially inflating detection performance.
- Baselines use heterogeneous architectures; comparing them directly may conflate architectural capacity with algorithmic novelty detection capability.
- PU-learning baselines require the exact OOD ratio in the unlabeled set; using estimated ratios degrades performance and breaks the upper-bound guarantee.

## Evidence (verbatim from paper)

> For a test sample, we average the outputs of the softmax probabilities predicted by the models, and use the entropy of the resulting distribution as the score for the hypothesis test described in Section 2.3. ... Therefore, we obtain an upper bound on the AUROC/TNR@95.

## Citation

```bibtex
@misc{tifrea2020semi,
  title={Semi-supervised novelty detection using ensembles with regularized disagreement},
  author={Tifrea et al. (2020)},
  year={2020},
  note={arXiv:2012.05825}
}
```

- arXiv: 2012.05825