Constrained Adaptive Attack Eval

---
name: constrained-adaptive-attack-eval
description: Evaluates the adversarial robustness of tabular deep learning models under realistic, domain-aware constraints. It measures how easily an attacker can flip model predictions while respecting feature mutability, boundaries, types, and relational constraints across ten progressively restricted threat models. Use when the user wants to benchmark on phishing, credit scoring, botnet detection, or asks about evaluating this task. Reports adversarial_label_flip.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.04503
  bibtex_key: simonetto2023constrained
  confidence: high
---

# constrained-adaptive-attack-eval

> Constrained Adaptive Attacks: Realistic Evaluation of Adversarial Examples and Robust Training of Deep Neural Networks for Tabular Data — Simonetto et al. (2023) (arXiv:2311.04503, 2023)

## What this evaluates

Evaluates the adversarial robustness of tabular deep learning models under realistic, domain-aware constraints. It measures how easily an attacker can flip model predictions while respecting feature mutability, boundaries, types, and relational constraints across ten progressively restricted threat models.

## Datasets

- **phishing** — total ?; splits: test (-1)
- **credit scoring** — total ?; splits: test (-1)
- **botnet detection** — total ?; splits: test (-1)

## Metrics

- `adversarial_label_flip` **(primary)** — range: [0, 1]
  - The fraction of test instances where the generated adversarial example satisfies all domain constraints and causes the model to predict a different label than the ground truth. Computed as the count of successful constrained label flips divided by the total test set size.

## Input / output format

**Input**: Original tabular feature vector x with ground truth label y, along with the threat model specification defining attacker knowledge (domain, model, dataset access).

**Output**: Perturbed feature vector x' = x + delta that satisfies domain constraints Omega and ideally flips the model's prediction.

## Scoring recipe

```python
successes = 0
for x, y in test_set:
    x_adv = generate_adversarial(x, y, threat_model)
    if satisfies_constraints(x_adv, Omega) and model.predict(x_adv) != y:
        successes += 1
metric_value = successes / len(test_set)
```

## Common pitfalls

- Applying image-based L_p perturbation bounds directly to tabular data without respecting categorical types or feature boundaries.
- Assuming white-box access for all threat models; the protocol explicitly defines 10 scenarios with varying knowledge levels (A1-E2).
- Ignoring feature relation constraints (e.g., open_accounts <= total_accounts), which renders adversarial examples unrealistic and easily detectable.

## Evidence (verbatim from paper)

> The adversary’s goal is to flip the label of a binary classification from the legitimate label to the adversary label. We investigate for the attacker three dimensions of capabilities of the target system, and we summarize our scenarios in Table 2... Each scenario A, B, C, D, and E has two variants.

## Citation

```bibtex
@misc{simonetto2023constrained,
  title={Constrained Adaptive Attacks: Realistic Evaluation of Adversarial Examples and Robust Training of Deep Neural Networks for Tabular Data},
  author={Simonetto et al. (2023)},
  year={2023},
  note={arXiv:2311.04503}
}
```

- arXiv: 2311.04503