Visat Robustness Eval

---
name: visat-robustness-eval
description: Evaluates the robustness of traffic sign recognition models against adversarial attacks (PGD) and distribution shifts (ImageNet-C corruptions, color quantization). It specifically probes multi-task learning models for spurious correlations across visual attributes (color, shape, symbol, text) by measuring error propagation and task-dependent vulnerability. Use when the user wants to benchmark on VISAT, or asks about evaluating this task. Reports epsilon (model error).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.26833
  bibtex_key: yu2025visat
  confidence: high
---

# visat-robustness-eval

> VISAT: Benchmarking Adversarial and Distribution Shift Robustness in Traffic Sign Recognition with Visual Attributes — Simon Yu et al. (arXiv:2510.26833, 2025)

## What this evaluates

Evaluates the robustness of traffic sign recognition models against adversarial attacks (PGD) and distribution shifts (ImageNet-C corruptions, color quantization). It specifically probes multi-task learning models for spurious correlations across visual attributes (color, shape, symbol, text) by measuring error propagation and task-dependent vulnerability.

## Datasets

- **VISAT** — total ?; splits: test (-1), test_attacked (-1)

## Metrics

- `epsilon (model error)` **(primary)** — range: [0, 1]
  - 1 - model accuracy. Represents the overall error rate on a given split.
- `Delta_epsilon (relative model error)` — range: [0, 1]
  - epsilon_at - epsilon_cl. The difference between model error on an attacked split and the clean split.
- `epsilon_cumulative` — range: [0, 1]
  - Sum of Delta_epsilon across all MTL tasks under a single attack. Measures total robustness degradation across tasks.
- `RECorr` — range: [0, 1]
  - (1/n) * sum_{i=0}^{n} (Delta_epsilon_i / Delta_epsilon_t), where n=3 non-targeted tasks and t is the targeted task. Measures error correlation propagation across MTL tasks.

## Input / output format

**Input**: Traffic sign image patches (cropped from MTSD). For MTL models, inputs are paired with visual attribute labels (color, shape, symbol, text).

**Output**: Predicted traffic sign class, plus predicted visual attributes (color, shape, symbol, text) for MTL models.

## Scoring recipe

```python
def compute_metrics(preds, labels, is_attacked=False):
    accuracy = mean(preds == labels)
    epsilon = 1.0 - accuracy
    if is_attacked:
        epsilon_at = epsilon
        delta_epsilon = epsilon_at - epsilon_cl
        # For MTL tasks:
        epsilon_cumulative = sum(delta_epsilon_i for i in tasks)
        re_corr = mean(delta_epsilon_i / delta_epsilon_target for i in non_targeted_tasks)
    return epsilon, delta_epsilon, epsilon_cumulative, re_corr
```

## Common pitfalls

- PGD attacks are highly model-specific; cross-examination shows attacks generated for ResNet-152 are ineffective against ViT-B/32, and vice versa.
- Blurring corruptions yield minimal performance drops because the dataset already contains small, distant sign patches that are naturally blurry, making blur an ineffective distribution shift.
- MTL models with disabled backbone fine-tuning (Linear heads) show uniform vulnerability across all tasks, as PGD attacks solely target the task heads rather than shared backbone features.

## Evidence (verbatim from paper)

> For the following benchmarks, we use the metrics defined below to gauge model robustness: * • $\epsilon$: model error, i.e., 1 - model accuracy. * • $\epsilon_{clean}$ or $\epsilon_{cl}$: model error evaluated on the original, clean VISAT testing split. * • $\epsilon_{attacked}$ or $\epsilon_{at}$: model error evaluated on an attacked VISAT testing split. * • $\Delta_{\epsilon}$\= $\epsilon_{at}$ - $\epsilon_{cl}$: relative model error, i.e., difference between attacked and clean model errors. * • $\epsilon_{cumulative}$ or $\epsilon_{cu}$: cumulative relative model error across MTL tasks under an attack. * • $\textit{RECorr}\=\frac{1}{n}\sum_{i\=0}^{n}\frac{\Delta_{\epsilon_{i}}}{\Delta_{\epsilon_{t}}}$: relative error correlation across MTL tasks under one attack targeting task $t$. $n$ is the number of non-targeted MTL tasks, in our case, 3. $\Delta_{\epsilon_{\text{t}}}$ is the relative model error of the targeted MTL task.

## Citation

```bibtex
@misc{yu2025visat,
  title={VISAT: Benchmarking Adversarial and Distribution Shift Robustness in Traffic Sign Recognition with Visual Attributes},
  author={Simon Yu et al.},
  year={2025},
  note={arXiv:2510.26833}
}
```

- arXiv: 2510.26833