Easyrobust Eval

---
name: easyrobust-eval
description: Evaluates the adversarial robustness and out-of-distribution (OOD) generalization of vision models on large-scale image classification benchmarks. It measures clean accuracy, robust accuracy against AutoAttack, and corruption error rates across multiple synthetic and real-world distribution shifts. Use when the user wants to benchmark on ImageNet, ImageNet-C, ImageNet-R, ImageNet-A, ImageNet-Sketch, Stylized-ImageNet, ObjectNet, ImageNet-V2, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.16975
  bibtex_key: mao2025easyrobust
  confidence: high
---

# easyrobust-eval

> EasyRobust: A Comprehensive and Easy-to-use Toolkit for Robust and Generalized Vision — Mao et al. (2025) (arXiv:2503.16975, 2025)

## What this evaluates

Evaluates the adversarial robustness and out-of-distribution (OOD) generalization of vision models on large-scale image classification benchmarks. It measures clean accuracy, robust accuracy against AutoAttack, and corruption error rates across multiple synthetic and real-world distribution shifts.

## Datasets

- **ImageNet** — total 50000; splits: val (50000)
- **ImageNet-C** — total ?; splits: test (-1)
- **ImageNet-R** — total ?; splits: test (-1)
- **ImageNet-A** — total ?; splits: test (-1)
- **ImageNet-Sketch** — total ?; splits: test (-1)
- **Stylized-ImageNet** — total ?; splits: test (-1)
- **ObjectNet** — total ?; splits: test (-1)
- **ImageNet-V2** — total ?; splits: test (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Fraction of correctly predicted class labels out of total samples. Reported as a percentage.
- `mCE` — range: percent
  - Mean Corruption Error: average error rate across all corruption types and severities on ImageNet-C. Lower values indicate better robustness.
- `AutoAttack accuracy` — range: percent
  - Top-1 accuracy on 5000 adversarial examples generated by AutoAttack, sampled from the ImageNet validation set via RobustBench.

## Input / output format

**Input**: RGB images resized to 256×256 and center cropped to 224×224.

**Output**: Class label predictions (top-1).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, corruption_data=None):
    acc = (predictions == gold_labels).mean() * 100
    if corruption_data is not None:
        errors = 1.0 - (corruption_data['preds'] == corruption_data['labels']).mean(axis=0)
        mce = errors.mean() * 100
    else:
        mce = None
    return acc, mce
```

## Common pitfalls

- mCE is a lower-is-better metric, whereas accuracy is higher-is-better; confusing the direction can invert conclusions.
- AutoAttack evaluation uses a fixed set of 5000 sampled images from RobustBench, not the full 50k validation set.
- Different methods use different training configurations (epochs, LR schedules, augmentations), so results are not directly comparable without matching training protocols.

## Evidence (verbatim from paper)

> Except for ImageNet-C which is measured by mCE, we report the top-1 accuracy on all used attacks and datasets. ... clean performance is the top@1 accuracy (%) on 50000 ImageNet validation images and robustness performance is the robust top@1 accuracy (%) on 5000 AutoAttack adversarial examples based on sampled images provided by RobustBench.

## Citation

```bibtex
@misc{mao2025easyrobust,
  title={EasyRobust: A Comprehensive and Easy-to-use Toolkit for Robust and Generalized Vision},
  author={Mao et al. (2025)},
  year={2025},
  note={arXiv:2503.16975}
}
```

- arXiv: 2503.16975