Vit Robustness Eval

---
name: vit-robustness-eval
description: Evaluates the robustness of Vision Transformer models against input perturbations including adversarial attacks (FGSM/PGD), spatial transformations, and restricted attention. It probes whether ViTs maintain classification performance under distribution shifts and targeted attacks compared to standard CNNs. Use when the user wants to benchmark on Unspecified, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.14586
  bibtex_key: [REDACTED]
  confidence: medium
---

# vit-robustness-eval

> Understanding Robustness of Transformers for Image Classification — Bhojanapalli et al. (2021) (arXiv:2103.14586, 2021)

## What this evaluates

Evaluates the robustness of Vision Transformer models against input perturbations including adversarial attacks (FGSM/PGD), spatial transformations, and restricted attention. It probes whether ViTs maintain classification performance under distribution shifts and targeted attacks compared to standard CNNs.

## Datasets

- **Unspecified** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified images out of the total test set. For robustness evaluations, computed as the accuracy on perturbed inputs (adversarial or spatially transformed).

## Input / output format

**Input**: RGB images resized to 384x384 without cropping, with intensities normalized to the model's expected range.

**Output**: Discrete class label prediction for each input image.

## Scoring recipe

```python
correct = 0
for img, label in test_set:
    pred = model(img)
    if pred == label:
        correct += 1
accuracy = correct / len(test_set)
```

## Common pitfalls

- Using standard center-crop preprocessing instead of the specified direct resize without cropping, which changes the evaluation distribution.
- Failing to clip adversarial perturbations to the valid pixel intensity range after each gradient step, leading to invalid inputs.
- Misinterpreting the spatial attack success condition; it requires checking if any transformation in the 2511-element grid causes misclassification, not just the maximum perturbation.

## Evidence (verbatim from paper)

> The attack is considered successful if any of the transformed images is incorrectly classified by the corresponding model model. The same fixed set of 2511(9×9×31) transformations was used for all images and all models.

## Citation

```bibtex
@misc{bhojanapalli2021understanding,
  title={Understanding Robustness of Transformers for Image Classification},
  author={Bhojanapalli et al. (2021)},
  year={2021},
  note={arXiv:2103.14586}
}
```

- arXiv: 2103.14586