Dialogue Safety Robustness Eval

---
name: dialogue-safety-robustness-eval
description: Evaluates the robustness of offensive language detection models against adversarial human attacks in single-turn and multi-turn dialogue contexts. It measures classifier resilience when exposed to iterative, context-aware attacks designed to evade safety filters. Use when the user wants to benchmark on Wikipedia Toxic Comments, or asks about evaluating this task. Reports Weighted-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1908.06083
  bibtex_key: dinan2019builditbreakit
  confidence: medium
---

# dialogue-safety-robustness-eval

> Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack — Dinan et al. (2019) (arXiv:1908.06083, 2019)

## What this evaluates

Evaluates the robustness of offensive language detection models against adversarial human attacks in single-turn and multi-turn dialogue contexts. It measures classifier resilience when exposed to iterative, context-aware attacks designed to evade safety filters.

## Datasets

- **Wikipedia Toxic Comments** — total ?; splits: test (-1)

## Metrics

- `Weighted-F1` **(primary)** — range: [0, 1]
  - Weighted average of F1 scores across classes (SAFE and OFFENSIVE), weighted by the number of true instances per class.
- `F1 (OFFENSIVE)` — range: [0, 1]
  - Harmonic mean of precision and recall calculated specifically for the OFFENSIVE class.

## Input / output format

**Input**: Dialogue context (single-turn or 4-turn multi-turn) containing a target utterance to be classified.

**Output**: Binary label: SAFE or OFFENSIVE.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    # Calculate TP, FP, FN for each class
    prec = TP / (TP + FP)
    rec = TP / (TP + FN)
    f1 = 2 * prec * rec / (prec + rec)
    # Weighted-F1: average F1 weighted by class support
    weighted_f1 = sum(f1[c] * support[c] for c in classes) / total_samples
    return weighted_f1, f1['OFFENSIVE']
```

## Common pitfalls

- Adversarial evaluation is round-dependent; performance varies significantly depending on which attack round (1, 2, or 3) generated the test data.
- Multi-turn evaluation uses a fixed 4-turn context window, which may not generalize to longer or more complex conversations.
- Human evaluation scores (0-5) measure attack success/quality but the exact annotation rubric is not specified in this section.

## Evidence (verbatim from paper)

> We report F1, precision, and recall for the OFFENSIVE class, as well as weighted-F1 for models $S_{i}$ and $A_{i}$ on the single-turn standard and adversarial tasks in Table 13.

## Citation

```bibtex
@misc{dinan2019builditbreakit,
  title={Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack},
  author={Dinan et al. (2019)},
  year={2019},
  note={arXiv:1908.06083}
}
```

- arXiv: 1908.06083