Tart Eval

---
name: tart-eval
description: This protocol evaluates a model's ability to maintain high accuracy on clean, unperturbed data while resisting adversarial attacks. It specifically probes the trade-off between clean accuracy and robustness under l_infinity perturbation constraints using both a custom simulated manifold dataset and the standard CIFAR-10 benchmark. Use when the user wants to benchmark on Transformed Hemisphere, CIFAR-10, or asks about evaluating this task. Reports Clean test accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.14728
  bibtex_key: yi2024tart
  confidence: high
---

# tart-eval

> TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training — Yi et al. (2024) (arXiv:2408.14728, 2024)

## What this evaluates

This protocol evaluates a model's ability to maintain high accuracy on clean, unperturbed data while resisting adversarial attacks. It specifically probes the trade-off between clean accuracy and robustness under l_infinity perturbation constraints using both a custom simulated manifold dataset and the standard CIFAR-10 benchmark.

## Datasets

- **Transformed Hemisphere** — total ?; splits: test (-1)
- **CIFAR-10** — total ?; splits: test (-1)

## Metrics

- `Clean test accuracy` **(primary)** — range: percent
  - Percentage of correctly classified samples on the original, unperturbed test set.
- `Robust test accuracy` — range: percent
  - Percentage of correctly classified samples on adversarially perturbed test sets, specifically evaluated against PGD^20 with l_infinity bound epsilon and step size alpha = epsilon/10.

## Input / output format

**Input**: Image tensors (CIFAR-10) or high-dimensional vectors (Transformed Hemisphere) representing natural data points.

**Output**: Predicted class label for each input instance.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- The 'Transformed Hemisphere' dataset is a custom simulated benchmark with no public release; results vary significantly with dimension d and class count c.
- Evaluation attacks (FGSM, PGD^20, PGD^40, AutoAttack) use a fixed l_infinity bound of epsilon = 8/255, but differ in step sizes and white/black-box settings; confusing training vs. evaluation attack parameters leads to invalid comparisons.
- The paper reports both 'best checkpoint' and 'last checkpoint' clean accuracy; selecting the wrong checkpoint can artificially inflate or deflate reported clean accuracy.

## Evidence (verbatim from paper)

> In Table II, we compare the performance of TART and Reverse-TART based on clean test accuracy and robust test accuracy. Clean test accuracy refers to the model accuracy on the test dataset without any attacks, while robust test accuracy is obtained by testing the accuracy against adversarial data generated by  $\mathrm{PGD^{20}}$  with random start,  $l_{\infty}$  perturbation bound  $\epsilon$ , and step size  $\alpha = \epsilon / 10$ .

## Citation

```bibtex
@misc{yi2024tart,
  title={TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training},
  author={Yi et al. (2024)},
  year={2024},
  note={arXiv:2408.14728}
}
```

- arXiv: 2408.14728