Robustbench Cifar10 Eval

---
name: robustbench-cifar10-eval
description: Evaluates adversarial robustness of image classifiers under multi-norm threat models, testing whether pre-screening diagnostics (FOSC, RDI) reliably predict full attack performance and expose worst-case vulnerabilities masked by single-norm evaluations. Use when the user wants to benchmark on RobustBench CIFAR-10, or asks about evaluating this task. Reports robust_accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20704
  bibtex_key: [REDACTED]
  confidence: high
---

# robustbench-cifar10-eval

> Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testing — Talluri (2026) (arXiv:2604.20704, 2026)

## What this evaluates

Evaluates adversarial robustness of image classifiers under multi-norm threat models, testing whether pre-screening diagnostics (FOSC, RDI) reliably predict full attack performance and expose worst-case vulnerabilities masked by single-norm evaluations.

## Datasets

- **RobustBench CIFAR-10** — total 50000; splits: test (10000); repo https://github.com/abhitall/auto-art

## Metrics

- `robust_accuracy` **(primary)** — range: percent
  - Percentage of correctly classified images under adversarial perturbations constrained within specified norm bounds (e.g., ℓ∞, ℓ2, ℓ1). Computed as correct predictions divided by total test samples.
- `kendall_tau` — range: [-1, 1]
  - Rank correlation coefficient measuring the agreement between diagnostic screening scores (RDI) and full attack robust accuracy rankings across models.
- `gradient_masking_detection_rate` — range: percent
  - Percentage of known gradient-masking configurations correctly flagged by the First-Order Stationarity Condition (FOSC) detector using a threshold of 0.1.

## Input / output format

**Input**: CIFAR-10 test images, model checkpoint weights, and attack configuration parameters (norm type, epsilon, step size, iteration count, batch size).

**Output**: Per-model robust accuracy scores across ℓ1, ℓ2, and ℓ∞ norms, FOSC/RDI diagnostic scores, and binary flags indicating gradient masking detection.

## Scoring recipe

```python
def compute_robust_accuracy(predictions, gold_labels):
    correct = (predictions == gold_labels).sum()
    return correct / len(gold_labels) * 100

def compute_kendall_tau(scores_a, scores_b):
    from scipy.stats import kendalltau
    return kendalltau(scores_a, scores_b).correlation

def compute_detection_rate(flags, ground_truth_masking):
    true_positives = sum(f & g for f, g in zip(flags, ground_truth_masking))
    return true_positives / len(ground_truth_masking) * 100
```

## Common pitfalls

- Single-norm (ℓ∞) evaluation overestimates robustness by up to 23.5 pp compared to worst-case multi-norm evaluation, masking catastrophic failures.
- RDI screening must be computed on exactly 500 samples with noise σ=0.01 to maintain the reported ~30× speedup and rank agreement.
- FOSC threshold τ_mask=0.1 is strict; models slightly above this threshold are flagged as having unreliable gradients, which can skew triage if relaxed.

## Evidence (verbatim from paper)

> The FOSC detector flags two models (FOSC >0.1), both with lower robust accuracy and RDI scores consistent with unreliable gradients. RDI rankings agree with full AutoAttack rankings: Kendall τ=0.82 (95% CI: 0.71–0.93), confirming that RDI provides a reliable ~30× faster proxy for model triage.

## Citation

```bibtex
@misc{talluri2026autoart,
  title={Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testing},
  author={Talluri (2026)},
  year={2026},
  note={arXiv:2604.20704}
}
```

- arXiv: 2604.20704