Wilds Eval

---
name: wilds-eval
description: Evaluates machine learning models' robustness to real-world distribution shifts, specifically domain generalization and subpopulation shifts. It measures how much model performance degrades when tested on out-of-distribution (OOD) data compared to in-distribution (ID) data, highlighting gaps in generalization for real-world deployment. Use when the user wants to benchmark on WILDS, or asks about evaluating this task. Reports ID and OOD performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.07421
  bibtex_key: koh2020wilds
  confidence: high
---

# wilds-eval

> WILDS: A Benchmark of in-the-Wild Distribution Shifts — Pang Wei Koh et al. (2020) (arXiv:2012.07421, 2020)

## What this evaluates

Evaluates machine learning models' robustness to real-world distribution shifts, specifically domain generalization and subpopulation shifts. It measures how much model performance degrades when tested on out-of-distribution (OOD) data compared to in-distribution (ID) data, highlighting gaps in generalization for real-world deployment.

## Datasets

- **WILDS** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/p-lambda/wilds

## Metrics

- `ID and OOD performance` **(primary)** — range: [0, 1]
  - Reports both in-distribution (ID) and out-of-distribution (OOD) performance, typically calculated as accuracy or error rate on the respective data splits.

## Input / output format

**Input**: Standardized dataset instances (e.g., medical images, satellite imagery, codebases) with ground-truth labels.

**Output**: Model predictions for the target task.

## Scoring recipe

```python
def compute_metrics(predictions, gold, id_mask):
    id_acc = (predictions[id_mask] == gold[id_mask]).mean()
    ood_acc = (predictions[~id_mask] == gold[~id_mask]).mean()
    return {'id_performance': id_acc, 'ood_performance': ood_acc}
```

## Common pitfalls

- Overfitting to the limited OOD test domains (e.g., single hospital in CAMELYON17-wILDS)
- Failing to report both ID and OOD performance separately
- Using external data or non-default architectures that confound algorithmic contributions

## Evidence (verbatim from paper)

> To better understand the extent to which any gains in OOD performance can be attributed to improved ID performance versus a model that is more robust to (i.e., less affected by) the distribution shift, we encourage model developers to report both ID and OOD performance numbers.

## Citation

```bibtex
@misc{koh2020wilds,
  title={WILDS: A Benchmark of in-the-Wild Distribution Shifts},
  author={Pang Wei Koh et al. (2020)},
  year={2020},
  note={arXiv:2012.07421}
}
```

- arXiv: 2012.07421