Datacomp Eval

---
name: datacomp-eval
description: Evaluates the zero-shot generalization capability of vision-language models across a diverse suite of image classification benchmarks. It measures how well pre-trained image-text alignment transfers to unseen downstream tasks without fine-tuning. Use when the user wants to benchmark on ImageNet, DataComp evaluation datasets, or asks about evaluating this task. Reports ImageNet.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.17425
  bibtex_key: fang2023datafilteringnetworks
  confidence: medium
---

# datacomp-eval

> Data Filtering Networks — Fang et al. (2023) (arXiv:2309.17425, 2023)

## What this evaluates

Evaluates the zero-shot generalization capability of vision-language models across a diverse suite of image classification benchmarks. It measures how well pre-trained image-text alignment transfers to unseen downstream tasks without fine-tuning.

## Datasets

- **ImageNet** — total ?; splits: test (-1)
- **DataComp evaluation datasets** — total ?; splits: test (-1)

## Metrics

- `ImageNet` **(primary)** — range: [0, 1]
  - Zero-shot classification accuracy on the ImageNet validation set, calculated as the percentage of correctly predicted class labels.
- `Average` — range: [0, 1]
  - Mean zero-shot classification accuracy aggregated across the 38 DataComp evaluation datasets.

## Input / output format

**Input**: Single images for zero-shot classification using text prompts.

**Output**: Predicted class labels for each image.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

imagenet_acc = compute_accuracy(imagenet_preds, imagenet_labels)
avg_acc = sum(compute_accuracy(preds, labels) for preds, labels in datacomp_datasets) / 38
```

## Common pitfalls

- The 'Average' metric masks domain-specific performance variations across the 38 datasets.
- Zero-shot accuracy is highly sensitive to the exact prompt templates used for classification, which are not specified in this section.

## Evidence (verbatim from paper)

> Below we provide links to checkpoints and detailed evaluation results of models in Table 3 on each of the 38 DataComp evaluation datasets

<table><tr><td>Model Link</td><td>ImageNet</td><td>Average</td></tr>...

## Citation

```bibtex
@misc{fang2023datafilteringnetworks,
  title={Data Filtering Networks},
  author={Fang et al. (2023)},
  year={2023},
  note={arXiv:2309.17425}
}
```

- arXiv: 2309.17425