Nips 2017 Adv Competition Eval

---
name: nips-2017-adv-competition-eval
description: Evaluates machine learning models' robustness against adversarial examples across three tracks: generating adversarial perturbations (non-targeted and targeted attacks) and defending against them. It measures both average performance and worst-case vulnerability to imperceptible perturbations. Use when the user wants to benchmark on NIPS 2017 Adversarial Attacks and Defences Competition dataset, or asks about evaluating this task. Reports Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1804.00097
  bibtex_key: kurakin2018adversarial
  confidence: high
---

# nips-2017-adv-competition-eval

> Adversarial Attacks and Defences Competition — Kurakin et al. (2018) (arXiv:1804.00097, 2018)

## What this evaluates

Evaluates machine learning models' robustness against adversarial examples across three tracks: generating adversarial perturbations (non-targeted and targeted attacks) and defending against them. It measures both average performance and worst-case vulnerability to imperceptible perturbations.

## Datasets

- **NIPS 2017 Adversarial Attacks and Defences Competition dataset** — total ?; splits: test (-1); repo https://github.com/cihangxie/NIPS2017_adv_challenge_defense

## Metrics

- `Score` **(primary)** — range: [0, 1]
  - Normalized success rate calculated as Raw Score divided by the total number of evaluated images. Ranges from 0 to 1.
- `Raw Score` — range: other
  - Un-normalized count of successful trials (e.g., number of images where the attack fooled the model or the defense correctly classified it).
- `Worst Score` — range: [0, 1]
  - The minimum Score achieved across different attack/defense combinations or perturbation budgets, indicating worst-case vulnerability.
- `Median eval time` — range: other
  - Median time required to evaluate one batch of 100 images, measuring computational efficiency.

## Input / output format

**Input**: Image batches (typically 100 images per batch) containing either clean images (for defense track) or adversarially perturbed images (for attack track).

**Output**: For attacks: adversarial perturbation added to the input image. For defenses: predicted class label for the input image.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, total_images):
    raw_score = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    score = raw_score / total_images
    worst_score = min(score_across_subsets)
    eval_time = median_time_per_batch(predictions, gold_labels)
    return {'score': score, 'raw_score': raw_score, 'worst_score': worst_score, 'median_eval_time': eval_time}
```

## Common pitfalls

- Confusing the normalized Score with the Raw Score, which is simply the un-normalized count of successful trials.
- Focusing only on average Score while ignoring Worst Score, which exposes models to catastrophic failure under specific adversarial conditions.
- Overlooking Median eval time, which is critical for assessing real-time robustness but often omitted in standard accuracy reports.

## Evidence (verbatim from paper)

> score is submission score as described in Section 3.3, raw score is un-normalized score which is number of times submission got a point on the image, worst score is submission score in the worst case and medial eval time is median time needed for evaluation of one batch of 100 images.

## Citation

```bibtex
@misc{kurakin2018adversarial,
  title={Adversarial Attacks and Defences Competition},
  author={Kurakin et al. (2018)},
  year={2018},
  note={arXiv:1804.00097}
}
```

- arXiv: 1804.00097