Clip Robustness Eval

---
name: clip-robustness-eval
description: Evaluates the zero-shot robustness of CLIP models to natural distribution shifts by measuring classification accuracy on four ImageNet-derived datasets. It probes how pre-training data composition and quality affect generalization to out-of-distribution images like sketches, renditions, and novel viewpoints. Use when the user wants to benchmark on ImageNet-V2, ImageNet-R, ImageNet-Sketch, ObjectNet, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2208.05516
  bibtex_key: nguyen2022quality
  confidence: high
---

# clip-robustness-eval

> Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP — Nguyen et al. (2022) (arXiv:2208.05516, 2022)

## What this evaluates

Evaluates the zero-shot robustness of CLIP models to natural distribution shifts by measuring classification accuracy on four ImageNet-derived datasets. It probes how pre-training data composition and quality affect generalization to out-of-distribution images like sketches, renditions, and novel viewpoints.

## Datasets

- **ImageNet-V2** — total ?; splits: test (-1)
- **ImageNet-R** — total ?; splits: test (-1)
- **ImageNet-Sketch** — total ?; splits: test (-1)
- **ObjectNet** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Top-1 zero-shot classification accuracy: the proportion of test images for which the model's highest similarity score matches the ground-truth class label.

## Input / output format

**Input**: An image and a set of text prompts corresponding to the ImageNet class names.

**Output**: Predicted class label (or class similarity scores) for each image.

## Scoring recipe

```python
correct = 0
total = 0
for image, gt_class in test_set:
    similarities = model.encode_image(image) @ model.encode_text(class_prompts).T
    pred_class = class_names[argmax(similarities)]
    if pred_class == gt_class:
        correct += 1
    total += 1
accuracy = correct / total
```

## Common pitfalls

- The evaluation measures zero-shot accuracy, not fine-tuned performance, so results should not be directly compared to fine-tuned baselines without clarification.
- ObjectNet only overlaps with 113 ImageNet classes, making its accuracy scores incomparable to ImageNet-V2, R, and Sketch which use 200 classes.

## Evidence (verbatim from paper)

> We vary the pre-training set size and hyperparameters such as number of epochs to obtain different accuracies on each data distribution. Similar to Taori et al. [75] and Radford et al. [61], we choose ImageNet as the reference distribution and evaluate CLIP on four natural distribution shifts derived from ImageNet: ImageNet-V2 [65]: A reproduction of the ImageNet validation set closely following the original dataset creation process. ImageNet-R [36]: Renditions (e.g., sculptures, paintings, etc.) for 200 ImageNet classes. ImageNet-Sketch [81]: Sketches of ImageNet class objects. ObjectNet [6]: A test set of objects in novel backgrounds, rotations, and viewpoints with 113 classes overlapping with ImageNet

## Citation

```bibtex
@misc{nguyen2022quality,
  title={Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP},
  author={Nguyen et al. (2022)},
  year={2022},
  note={arXiv:2208.05516}
}
```

- arXiv: 2208.05516