Medical Ood Detection Eval

---
name: medical-ood-detection-eval
description: Evaluates the ability of out-of-distribution detection (OoDD) methods to correctly classify medical images as in-distribution or out-of-distribution across three distinct OoD categories: unrelated domains, incorrect image preparation, and unseen medical conditions due to selection bias. Use when the user wants to benchmark on Medical OoD Benchmark ($D_{test}$), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2007.04250
  bibtex_key: cao2020benchmarkmedicalood
  confidence: high
---

# medical-ood-detection-eval

> A Benchmark of Medical Out of Distribution Detection — Cao et al. (2020) (arXiv:2007.04250, 2020)

## What this evaluates

Evaluates the ability of out-of-distribution detection (OoDD) methods to correctly classify medical images as in-distribution or out-of-distribution across three distinct OoD categories: unrelated domains, incorrect image preparation, and unseen medical conditions due to selection bias.

## Datasets

- **Medical OoD Benchmark ($D_{test}$)** — total ?; splits: test (-1), val (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified In and Out samples on the class-balanced test set. Provides an unbiased representation of type I and type II errors.
- `AUPRC` — range: [0, 1]
  - Area Under the Precision-Recall Curve. Characterizes the separability of In and Out samples based on predicted scores, which are thresholded to obtain binary classifications.

## Input / output format

**Input**: Medical images (e.g., chest X-rays, fundus images, histology slides) processed by OoDD methods to generate In/Out scores or classifications.

**Output**: Binary classification predictions (In vs. Out) or continuous OoDD scores, thresholded to yield final predictions.

## Scoring recipe

```python
def compute_metrics(predictions, gold, scores):
    # predictions and gold are binary (0=In, 1=Out)
    accuracy = (predictions == gold).mean()
    # AUPRC requires continuous scores and binary gold labels
    precision, recall, _ = precision_recall_curve(gold, scores)
    auprc = auc(recall, precision)
    return accuracy, auprc
```

## Common pitfalls

- Using a single Out dataset for validation ($D_{val}$) instead of randomly sampling multiple Out datasets, which reduces generalization and decision boundary stability.
- Assuming complex auxiliary models outperform simple classifier-only baselines; the benchmark shows simple methods often match or exceed complex ones.
- Underestimating the difficulty of Use-Case 3 (unseen diseases/selection bias), where methods perform near random chance.

## Evidence (verbatim from paper)

> We measure the accuracy and Area Under Precision-Recall Curve (AUPRC) on  $D_{test}$  , totaling at 11 pairs of performance numbers per method. Since  $D_{test}$  is class-balanced, accuracy provides an unbiased representation of type I and type II errors. AUPRC characterizes the separability of  $In$  and  $Out$  samples in predicted value (the value that we threshold to obtain classification).

## Citation

```bibtex
@misc{cao2020benchmarkmedicalood,
  title={A Benchmark of Medical Out of Distribution Detection},
  author={Cao et al. (2020)},
  year={2020},
  note={arXiv:2007.04250}
}
```

- arXiv: 2007.04250