Multirobustbench Eval

---
name: multirobustbench-eval
description: Evaluates machine learning model robustness against multiple diverse adversarial attacks (e.g., ℓₚ-norm, color shifts, spatial transformations) across varying strengths. It quantifies how well defenses maintain performance under worst-case and average-case multiattack scenarios, addressing bias from varying attack difficulties. Use when the user wants to benchmark on MultiRobustBench, or asks about evaluating this task. Reports competitiveness ratio (CR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.10980
  bibtex_key: dai2023multirobustbench
  confidence: high
---

# multirobustbench-eval

> MultiRobustBench: Benchmarking Robustness Against Multiple Attacks — Dai et al. (2023) (arXiv:2302.10980, 2023)

## What this evaluates

Evaluates machine learning model robustness against multiple diverse adversarial attacks (e.g., ℓₚ-norm, color shifts, spatial transformations) across varying strengths. It quantifies how well defenses maintain performance under worst-case and average-case multiattack scenarios, addressing bias from varying attack difficulties.

## Datasets

- **MultiRobustBench** — total ?; splits: test (-1)

## Metrics

- `competitiveness ratio (CR)` **(primary)** — range: [0, 1]
  - Computes robust accuracy for each attack type and strength, then weights it by the inverse of the best achievable accuracy for that attack (1/acc*(P)) to correct for difficulty bias. Aggregates these weighted accuracies to report average-case (CR_ind-avg) and worst-case (CR_ind-worst) multiattack robustness.

## Input / output format

**Input**: Images subjected to various adversarial attacks (e.g., ℓₚ-norm perturbations, color changes, spatial transformations) at specified attack strengths.

**Output**: Predicted class labels for the perturbed images.

## Scoring recipe

```python
# For each attack type P in the 9 evaluated attacks:
#   acc_star_P = best achievable accuracy for P
#   weighted_accs = []
#   for each strength epsilon in 20 levels:
#       robust_acc = accuracy(model, attack(P, epsilon))
#       weighted_accs.append(robust_acc / acc_star_P)
# CR_ind_avg = mean(weighted_accs) across all attacks
# CR_ind_worst = min(weighted_accs) across all attacks
```

## Common pitfalls

- Reporting accuracy on a union of attacks or average accuracy without weighting by attack difficulty, which biases scores toward easier attacks.
- Evaluating only a single attack strength per attack type, which fails to capture robustness across the full threat spectrum.
- Ignoring the inherent difficulty of different attack types, leading to unfair comparisons between defenses.

## Evidence (verbatim from paper)

> we evaluate 9 different attack types with 20 levels of attack strength leading to a larger scope in evaluation. ... Our metric $\text{CR}_{\text{ind-worst}}$ addresses this bias by weighting the robust accuracy of the defense by $\frac{1}{\text{acc}^{*}(P)}$. Our metric $\text{CR}_{\text{ind-avg}}$ addresses this bias by weighting the robust accuracy of the defense by $\frac{1}{\text{acc}^{*}(P)}.

## Citation

```bibtex
@misc{dai2023multirobustbench,
  title={MultiRobustBench: Benchmarking Robustness Against Multiple Attacks},
  author={Dai et al. (2023)},
  year={2023},
  note={arXiv:2302.10980}
}
```

- arXiv: 2302.10980