Toxic Comment Eval

---
name: toxic-comment-eval
description: This benchmark evaluates the individual and group fairness of toxicity classifiers on online comments. It probes whether model predictions remain stable when sensitive identity tokens are swapped (individual fairness) and whether prediction accuracy is equitable across different demographic groups (group fairness). Use when the user wants to benchmark on Toxic Comment Classification Challenge, or asks about evaluating this task. Reports Balanced Accuracy (BA).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2006.14168
  bibtex_key: yurochkin2020sensi
  confidence: high
---

# toxic-comment-eval

> SenSeI: Sensitive Set Invariance for Enforcing Individual Fairness — Yurochkin et al. (2020) (arXiv:2006.14168, 2020)

## What this evaluates

This benchmark evaluates the individual and group fairness of toxicity classifiers on online comments. It probes whether model predictions remain stable when sensitive identity tokens are swapped (individual fairness) and whether prediction accuracy is equitable across different demographic groups (group fairness).

## Datasets

- **Toxic Comment Classification Challenge** — total ?; splits: train (-1), test (-1); repo https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge

## Metrics

- `Balanced Accuracy (BA)` **(primary)** — range: [0, 1]
  - Average of sensitivity and specificity, used to handle class imbalance.
- `Counterfactual Token Fairness (CTF)` — range: [0, 1]
  - Variance across 50 counterfactuals of the predicted probability that a comment is toxic.
- `Prediction Consistency (PC)` — range: [0, 1]
  - Proportion of test comments where the binary prediction remains identical across all 50 counterfactual variations.
- `Accuracy Parity STD` — range: [0, 1]
  - Standard deviation of balanced accuracies across 9 protected identity groups.

## Input / output format

**Input**: BERT (base, uncased) fine-tuned feature representations of online comments.

**Output**: Binary toxicity prediction (probability or logit).

## Scoring recipe

```python
For each test comment, generate 50 counterfactuals by swapping identity tokens.
ctf = variance(predicted_probabilities)
pc = mean(predictions == original_predictions)
accuracy_parity_std = std(balanced_accuracy_per_group)
```

## Common pitfalls

- Counterfactual generation relies on a fixed set of 50 tokens, which may not cover all identity variations.
- Fairness metrics are evaluated on a held-out test set, but the fair metric used for training is learned via factor analysis on counterfactuals, potentially leaking information.

## Evidence (verbatim from paper)

> We report Counterfactual Token Fairness (CTF) score (Garg et al., 2018) that quantifies variance across counterfactuals of the predicted probability that a comment is toxic, and Prediction Consistency (PC) equal to the portion of test comments where prediction is the same across all 50 counterfactual variations.

## Citation

```bibtex
@misc{yurochkin2020sensi,
  title={SenSeI: Sensitive Set Invariance for Enforcing Individual Fairness},
  author={Yurochkin et al. (2020)},
  year={2020},
  note={arXiv:2006.14168}
}
```

- arXiv: 2006.14168