Zero Shot Transfer Eval

---
name: zero-shot-transfer-eval
description: This evaluation probes a vision-language model's ability to generalize to unseen image classification tasks without task-specific fine-tuning. It measures how well the model aligns visual features with natural language class descriptions to perform zero-shot classification across diverse domains. Use when the user wants to benchmark on ImageNet, CIFAR-10, Oxford-IIIT Pet, Food101, Stanford Cars, Kinetics700, EuroSAT, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.00020
  bibtex_key: radford2021clip
  confidence: high
---

# zero-shot-transfer-eval

> Learning Transferable Visual Models From Natural Language Supervision — Alec Radford et al. (arXiv:2103.00020, 2021)

## What this evaluates

This evaluation probes a vision-language model's ability to generalize to unseen image classification tasks without task-specific fine-tuning. It measures how well the model aligns visual features with natural language class descriptions to perform zero-shot classification across diverse domains.

## Datasets

- **ImageNet** — total ?; splits: test (-1)
- **CIFAR-10** — total ?; splits: test (-1)
- **Oxford-IIIT Pet** — total ?; splits: test (-1)
- **Food101** — total ?; splits: test (-1)
- **Stanford Cars** — total ?; splits: test (-1)
- **Kinetics700** — total ?; splits: test (-1)
- **EuroSAT** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Top-1 accuracy is the fraction of images where the predicted class matches the ground truth label. Top-5 accuracy is the fraction where the ground truth is among the top 5 predicted classes. Predictions are derived from softmax probabilities over cosine similarities between image and text embeddings.

## Input / output format

**Input**: An input image and a list of text labels representing all possible class names for the target dataset.

**Output**: A predicted class label (or a probability distribution over the provided class names).

## Scoring recipe

```python
def zero_shot_eval(images, class_names, ground_truths, temperature=0.07):
    preds = []
    for img, gt in zip(images, ground_truths):
        img_emb = encode_image(img)
        text_embs = [encode_text(c) for c in class_names]
        sims = [cosine(img_emb, e) / temperature for e in text_embs]
        pred_idx = argmax(softmax(sims))
        preds.append(class_names[pred_idx])
    return sum(1 for p, g in zip(preds, ground_truths) if p == g) / len(ground_truths)
```

## Common pitfalls

- Using raw class names without prompt templates (e.g., 'A photo of a {label}.') significantly hurts performance due to distribution mismatch with pre-training data.
- Polysemous class names (e.g., 'crane', 'boxer') confuse the text encoder, leading to incorrect zero-shot predictions.
- Many standard datasets lack proper text-to-label mappings or use arbitrary numeric IDs, making zero-shot transfer impossible without manual curation.

## Evidence (verbatim from paper)

> The best CLIP model improves accuracy on ImageNet from a proof of concept 11.5% to 76.2% and matches the performance of the original ResNet-50 despite using none of the 1.28 million crowd-labeled training examples available for this dataset. Additionally, the top-5 accuracy of CLIP models are noticeably higher than their top-1, and this model has a 95% top-5 accuracy, matching Inception-V4.

## Citation

```bibtex
@misc{radford2021clip,
  title={Learning Transferable Visual Models From Natural Language Supervision},
  author={Alec Radford et al.},
  year={2021},
  note={arXiv:2103.00020}
}
```

- arXiv: 2103.00020