Vision Language Ood Eval

---
name: vision-language-ood-eval
description: Probes the ability of vision-language models to distinguish in-distribution from out-of-distribution samples under semantic, covariate, and real-world distribution shifts. It evaluates both zero-shot and few-shot prompt learning approaches across multiple benchmarks to assess robustness and ranking consistency. Use when the user wants to benchmark on ImageNet-X, ImageNet-FS-X, Wilds-FS-X, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.18463
  bibtex_key: noda2025oodx
  confidence: high
---

# vision-language-ood-eval

> A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models — Noda et al. (2025) (arXiv:2501.18463, 2025)

## What this evaluates

Probes the ability of vision-language models to distinguish in-distribution from out-of-distribution samples under semantic, covariate, and real-world distribution shifts. It evaluates both zero-shot and few-shot prompt learning approaches across multiple benchmarks to assess robustness and ranking consistency.

## Datasets

- **ImageNet-X** — total ?; splits: test (-1)
- **ImageNet-FS-X** — total ?; splits: test (-1)
- **Wilds-FS-X** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve. Measures the probability that a randomly chosen OOD sample receives a higher OOD score than a randomly chosen ID sample. Higher values indicate better separation, with 50% representing random guessing.

## Input / output format

**Input**: Image paired with text prompts (ID class labels and optionally OOD prompts) fed into a vision-language model (e.g., CLIP) to compute similarity scores.

**Output**: Per-image OOD score (scalar) derived from model similarity outputs, used to rank samples as ID or OOD.

## Scoring recipe

```python
def compute_auroc(predictions, gold):
    # predictions: array of OOD scores (higher = more OOD)
    # gold: binary array (1 for ID, 0 for OOD)
    fpr, tpr, _ = roc_curve(gold, predictions)
    return auc(fpr, tpr)
```

## Common pitfalls

- Assuming higher classification accuracy on ID data directly translates to better OOD detection performance, which the paper explicitly shows is not always true.
- Evaluating only on semantic shifts while ignoring covariate shifts, which causes significant and consistent performance drops in CLIP-based methods.
- Treating all OOD datasets as a single homogeneous group without distinguishing between common-OOD and hard-OOD, as method rankings shift significantly between them.

## Evidence (verbatim from paper)

> For evaluation, we use the area under the receiver operating characteristic curve (AUROC). A higher value indicates better performance, with 50% serving as the baseline.

## Citation

```bibtex
@misc{noda2025oodx,
  title={A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models},
  author={Noda et al. (2025)},
  year={2025},
  note={arXiv:2501.18463}
}
```

- arXiv: 2501.18463