Image Classification Eval

---
name: image-classification-eval
description: Evaluates the ability of vision models to learn transferable visual representations and perform accurate image classification across varying data scales and domain shifts. It probes how well patch-based self-attention architectures generalize from large-scale pre-training to standard and low-data downstream recognition tasks. Use when the user wants to benchmark on ImageNet (ILSVRC-2012), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.11929
  bibtex_key: dosovitskiy2020vit
  confidence: high
---

# image-classification-eval

> An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — Dosovitskiy et al. (2020) (arXiv:2010.11929, 2020)

## What this evaluates

Evaluates the ability of vision models to learn transferable visual representations and perform accurate image classification across varying data scales and domain shifts. It probes how well patch-based self-attention architectures generalize from large-scale pre-training to standard and low-data downstream recognition tasks.

## Datasets

- **ImageNet (ILSVRC-2012)** — total 1300000; splits: val (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Top-1 classification accuracy, calculated as the number of correctly predicted labels divided by the total number of test instances.

## Input / output format

**Input**: RGB images resized or cropped to a fixed resolution (e.g., 224x224, 384x384, or 512x512), divided into non-overlapping 16x16 patches, linearly projected, and passed through a Transformer encoder.

**Output**: A single predicted class label (or class probabilities) corresponding to the image content.

## Scoring recipe

```python
correct = 0
total = 0
for image, label in test_loader:
    logits = model(image)
    pred = argmax(logits, dim=1)
    correct += (pred == label).sum().item()
    total += label.size(0)
accuracy = correct / total
```

## Common pitfalls

- Confusing the pre-training dataset size (JFT-300M, ImageNet-21k, or ImageNet) with the evaluation dataset, which drastically changes reported accuracy.
- Overlooking that fine-tuning is performed at higher resolutions (512/518) for the largest models, which boosts accuracy compared to standard 224x224 evaluation.
- Mixing up fine-tuning accuracy with linear few-shot accuracy, which uses a frozen backbone and closed-form least-squares regression.

## Evidence (verbatim from paper)

> Metrics. We report results on downstream datasets either through few-shot or fine-tuning accuracy. Fine-tuning accuracies capture the performance of each model after fine-tuning it on the respective dataset. Few-shot accuracies are obtained by solving a regularized least-squares regression problem that maps the (frozen) representation of a subset of training images to {-1,1}^K target vectors.

## Citation

```bibtex
@misc{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy et al. (2020)},
  year={2020},
  note={arXiv:2010.11929}
}
```

- arXiv: 2010.11929