Toxicity Detection Eval

---
name: toxicity-detection-eval
description: Probes the ability of text generation models to produce non-toxic content by measuring average toxicity scores and comparing them via statistical significance testing. It specifically evaluates how accounting for classifier uncertainty affects the reliability of these comparisons. Use when the user wants to benchmark on BOLD, RealToxicityPrompts, or asks about evaluating this task. Reports Confidence Interval.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.17254
  bibtex_key: [REDACTED]
  confidence: high
---

# toxicity-detection-eval

> Faithful Model Evaluation for Model-Based Metrics — Goyal et al. (2023) (arXiv:2312.17254, 2023)

## What this evaluates

Probes the ability of text generation models to produce non-toxic content by measuring average toxicity scores and comparing them via statistical significance testing. It specifically evaluates how accounting for classifier uncertainty affects the reliability of these comparisons.

## Datasets

- **BOLD** — total 23679; splits: test (23679)
- **RealToxicityPrompts** — total 100000; splits: test (100000)

## Metrics

- `Confidence Interval` **(primary)** — range: [-1, 1]
  - Calculated as ATE ± 1.96 * sqrt(corrected_variance), where corrected_variance incorporates the metric model's prediction error (precision and false omission rate) rather than just sampling variance.

## Input / output format

**Input**: Prompts from BOLD or RealToxicityPrompts fed to a text generation model (GPT-2 or GPT-Neo) to produce responses, which are then scored by a toxicity classifier (RoBERTa-ToxiGen).

**Output**: Aggregated statistics: mean toxicity score, Average Treatment Effect (ATE), deterministic variance, corrected variance, and confidence intervals.

## Scoring recipe

```python
# Calculate mean toxicity for each model
mean_T = sum(toxicity_scores_T) / len(toxicity_scores_T)
mean_B = sum(toxicity_scores_B) / len(toxicity_scores_B)
# Calculate Average Treatment Effect
ate = mean_T - mean_B
# Calculate corrected variance incorporating classifier error (precision, FOR)
var_corrected = var_deterministic + correction_term(precision, FOR, n)
# Calculate 95% Confidence Interval
ci_lower = ate - 1.96 * math.sqrt(var_corrected)
ci_upper = ate + 1.96 * math.sqrt(var_corrected)
```

## Common pitfalls

- Ignoring the variance introduced by the metric model (classifier) leads to artificially narrow confidence intervals and false statistical significance.
- Treating the classifier's output as ground truth rather than an estimate with known precision and false omission rate.

## Evidence (verbatim from paper)

> ATE is calculated as the difference between average toxicity score of the two models, specifically, it is the average toxicity score of GPT-Neo subtracted by the average toxicity score of GPT2 (we consider GPT2 as baseline). Disregarding the metric model errors, the confidence interval is (-0.00325, -0.00114), leading to the conclusion that we can reject the null hypothesis and reaching the conclusion that GPT-Neo produces output with significant lower toxicity than GPT2. However, when we consider the metric model errors, the confidence interval is (-0.00978, 0.00538), which shows insignificant difference and we cannot reject null hypothesis.

## Citation

```bibtex
@misc{goyal2023faithful,
  title={Faithful Model Evaluation for Model-Based Metrics},
  author={Goyal et al. (2023)},
  year={2023},
  note={arXiv:2312.17254}
}
```

- arXiv: 2312.17254