Datacomp Zero Shot Eval

---
name: datacomp-zero-shot-eval
description: Evaluates zero-shot generalization of vision-language models across a broad suite of classification and retrieval tasks, with a focus on image-text alignment and retrieval accuracy. Use when the user wants to benchmark on DataComp Zero-Shot Suite, or asks about evaluating this task. Reports ImageNet accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.10350
  bibtex_key: nguyen2023improvingmultimodal
  confidence: high
---

# datacomp-zero-shot-eval

> Improving Multimodal Datasets with Image Captioning — Nguyen et al. (2023) (arXiv:2307.10350, 2023)

## What this evaluates

Evaluates zero-shot generalization of vision-language models across a broad suite of classification and retrieval tasks, with a focus on image-text alignment and retrieval accuracy.

## Datasets

- **DataComp Zero-Shot Suite** — total ?; splits: test (-1)

## Metrics

- `ImageNet accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly classified images in the zero-shot setting.
- `Average accuracy over 38 tasks` — range: [0, 1]
  - Mean accuracy across 38 classification and retrieval tasks from the DataComp benchmark.
- `Flickr30K/MS-COCO retrieval score` — range: [0, 1]
  - Average of text-to-image Recall@1 and image-to-text Recall@1.

## Input / output format

**Input**: For training: image-text pairs with synthetic captions. For evaluation: zero-shot images and text queries from benchmark datasets.

**Output**: Predicted class labels for classification tasks; ranked lists of images or texts for retrieval tasks.

## Scoring recipe

```python
# Classification accuracy
preds = model.zero_shot_predict(images)
acc = (preds == gold_labels).mean()

# Retrieval score (Flickr30K / MS-COCO)
r1_t2i = recall_at_k(gold_texts, candidate_images, k=1)
r1_i2t = recall_at_k(gold_images, candidate_texts, k=1)
retrieval_score = (r1_t2i + r1_i2t) / 2

# Overall suite metric
avg_acc = mean([acc_task for task in tasks_38])
```

## Common pitfalls

- Standard captioning metrics like CIDEr do not correlate with caption quality for CLIP training, so they should not be used to judge dataset utility.
- Retrieval performance must be averaged across both text-to-image and image-to-text Recall@1 directions, not reported for a single direction.
- Evaluation is strictly zero-shot; fine-tuning on benchmark data violates the protocol.

## Evidence (verbatim from paper)

> We adopt DataComp's zero-shot evaluation suite and report both ImageNet accuracy and the average accuracy over 38 classification and retrieval tasks proposed by the benchmark [18]. We also pay particular attention to retrieval performance on Flickr30K [54] and MS-COCO [11]. The retrieval score reported is the average of text-to-image Recall@1 and image-to-text Recall@1.

## Citation

```bibtex
@misc{nguyen2023improvingmultimodal,
  title={Improving Multimodal Datasets with Image Captioning},
  author={Nguyen et al. (2023)},
  year={2023},
  note={arXiv:2307.10350}
}
```

- arXiv: 2307.10350