Ood Detection Histopathology Eval

---
name: ood-detection-histopathology-eval
description: Evaluates the ability of uncertainty estimation methods to distinguish in-distribution clinical histopathology samples from out-of-distribution samples and to provide well-calibrated confidence scores for selective prediction. It probes how different methods maintain predictive accuracy and calibration under varying degrees of data shift. Use when the user wants to benchmark on CIFAR-10, Hospital 4, Hospital 5, Cancer type, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.09909
  bibtex_key: goetz2022uncertainty
  confidence: high
---

# ood-detection-histopathology-eval

> Uncertainty estimation for out-of-distribution detection in computational histopathology — Goetz (2022) (arXiv:2210.09909, 2022)

## What this evaluates

Evaluates the ability of uncertainty estimation methods to distinguish in-distribution clinical histopathology samples from out-of-distribution samples and to provide well-calibrated confidence scores for selective prediction. It probes how different methods maintain predictive accuracy and calibration under varying degrees of data shift.

## Datasets

- **CIFAR-10** — total ?; splits: train (-1), test (-1)
- **Hospital 4** — total ?; splits: test (-1)
- **Hospital 5** — total ?; splits: test (-1)
- **Cancer type** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly classified samples among those retained after uncertainty thresholding.
- `Youden's J` — range: [-1, 1]
  - Threshold optimization criterion for selective prediction, calculated as sensitivity + specificity - 1.
- `average precision` — range: [0, 1]
  - Area under the precision-recall curve, summarizing model performance across all classification thresholds.

## Input / output format

**Input**: Image patches (histopathology slides) or standard images (CIFAR-10) passed through a neural network feature extractor.

**Output**: Predicted class label, class probability distribution, and a scalar uncertainty score (e.g., predictive variance or distance-based uncertainty).

## Scoring recipe

```python
def compute_metrics(predictions, labels, uncertainty_scores, threshold):
    accepted_mask = uncertainty_scores < threshold
    acc = mean(predictions[accepted_mask] == labels[accepted_mask])
    retention_rate = mean(accepted_mask)
    return acc, retention_rate

# Threshold optimization via Youden's J
# J = sensitivity + specificity - 1
# threshold = argmax_J(J) on validation set
```

## Common pitfalls

- Setting uncertainty thresholds on the same data used for evaluation, which causes data leakage and inflates OOD detection performance.
- Assuming far OOD detection is inherently easier than near OOD detection; performance degrades sharply as data shift increases.
- Ignoring model calibration; high accuracy does not guarantee reliable uncertainty estimates for clinical deployment.

## Evidence (verbatim from paper)

> Table 4: Fraction of samples (mean ± std) retained after rejection based on an uncertainty threshold that optimized Youden's J, set on hospital 5 and hospital 4 for hospital 4 and hospital 5 & cancer type, respectively. ... While uncertainty thresholding does in general increase accuracy, the gains are very variable across datasets and methods.

## Citation

```bibtex
@misc{goetz2022uncertainty,
  title={Uncertainty estimation for out-of-distribution detection in computational histopathology},
  author={Goetz (2022)},
  year={2022},
  note={arXiv:2210.09909}
}
```

- arXiv: 2210.09909