Evaluates the zero-shot generalization capability of vision-language models across a diverse suite of image classification benchmarks. It measures how well pre-trained image-text alignment transfers to unseen downstream tasks without fine-tuning. Use when the user wants to benchmark on ImageNet, DataComp evaluation datasets, or asks about evaluating this task. Reports ImageNet.
Installs into .claude/skills of the current project.
Are you the author of Datacomp Eval?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/qhjqhj00-datacomp-eval)
---
name: datacomp-eval
description: Evaluates the zero-shot generalization capability of vision-language models across a diverse suite of image classification benchmarks. It measures how well pre-trained image-text alignment transfers to unseen downstream tasks without fine-tuning. Use when the user wants to benchmark on ImageNet, DataComp evaluation datasets, or asks about evaluating this task. Reports ImageNet.
metadata:
skill_kind: dataset_eval
source_arxiv: 2309.17425
bibtex_key: [REDACTED]
confidence: medium
---
# datacomp-eval
> Data Filtering Networks — Fang et al. (2023) (arXiv:2309.17425, 2023)
## What this evaluates
Evaluates the zero-shot generalization capability of vision-language models across a diverse suite of image classification benchmarks. It measures how well pre-trained image-text alignment transfers to unseen downstream tasks without fine-tuning.
## Datasets
- **ImageNet** — total ?; splits: test (-1)
- **DataComp evaluation datasets** — total ?; splits: test (-1)
## Metrics
- `ImageNet` **(primary)** — range: [0, 1]
- Zero-shot classification accuracy on the ImageNet validation set, calculated as the percentage of correctly predicted class labels.
- `Average` — range: [0, 1]
- Mean zero-shot classification accuracy aggregated across the 38 DataComp evaluation datasets.
## Input / output format
**Input**: Single images for zero-shot classification using text prompts.
**Output**: Predicted class labels for each image.
## Scoring recipe
```python
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return correct / len(gold_labels)
imagenet_acc = compute_accuracy(imagenet_preds, imagenet_labels)
avg_acc = sum(compute_accuracy(preds, labels) for preds, labels in datacomp_datasets) / 38
```
## Common pitfalls
- The 'Average' metric masks domain-specific performance variations across the 38 datasets.
- Zero-shot accuracy is highly sensitive to the exact prompt templates used for classification, which are not specified in this section.
## Evidence (verbatim from paper)
> Below we provide links to checkpoints and detailed evaluation results of models in Table 3 on each of the 38 DataComp evaluation datasets
<table><tr><td>Model Link</td><td>ImageNet</td><td>Average</td></tr>...
## Citation
```bibtex
@misc{fang2023datafilteringnetworks,
title={Data Filtering Networks},
author={Fang et al. (2023)},
year={2023},
note={arXiv:2309.17425}
}
```
- arXiv: 2309.17425
