Toxicity Perspectives Eval

---
name: toxicity-perspectives-eval
description: Evaluates how well automated toxicity classifiers align with diverse human perceptions of harmful content, specifically measuring how demographic background and personal harassment experiences influence toxicity judgments. Use when the user wants to benchmark on Toxicity Perspectives Dataset, or asks about evaluating this task. Reports interrater agreement (Cohen's kappa).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.04511
  bibtex_key: kumar2021designing
  confidence: high
---

# toxicity-perspectives-eval

> Designing Toxic Content Classification for a Diversity of Perspectives — Kumar et al. (2021) (arXiv:2106.04511, 2021)

## What this evaluates

Evaluates how well automated toxicity classifiers align with diverse human perceptions of harmful content, specifically measuring how demographic background and personal harassment experiences influence toxicity judgments.

## Datasets

- **Toxicity Perspectives Dataset** — total 107620; splits: full (107620)

## Metrics

- `interrater agreement (Cohen's kappa)` **(primary)** — range: [-1, 1]
  - Cohen's kappa coefficient measuring the agreement between two independent raters on categorical ratings, adjusted for chance. Used to validate coding consistency and measure convergence of human ratings to a global average toxicity score.
- `classification accuracy` — range: [0, 1]
  - Fraction of comments correctly classified as toxic or not toxic when comparing model predictions to human labels, evaluated with fixed or personalized thresholds per demographic cohort.

## Input / output format

**Input**: A social media comment text (sourced from Twitter, Reddit, or 4chan).

**Output**: Human ratings on a 5-point Likert scale ('Not at all toxic' to 'Extremely toxic'); model outputs a toxicity score between 0 and 1.

## Scoring recipe

```python
def compute_kappa(rater1_labels, rater2_labels):
    N = len(rater1_labels)
    Po = sum(1 for a, b in zip(rater1_labels, rater2_labels) if a == b) / N
    Pe = sum((count_r1[i] / N) * (count_r2[i] / N) for i in range(num_categories))
    return (Po - Pe) / (1 - Pe) if (1 - Pe) > 0 else 0

def compute_accuracy(predictions, human_labels, threshold):
    correct = sum(1 for p, h in zip(predictions, human_labels) if (p >= threshold) == (h >= threshold))
    return correct / len(predictions)
```

## Common pitfalls

- Assuming a single global toxicity threshold applies to all users; the study shows thresholds must be personalized per demographic or experiential cohort.
- Treating the 5-point Likert scale as a single binary label without accounting for the specific cutoff ('Moderately toxic' or greater) used to define the positive class.
- Ignoring the stratified sampling bias: the dataset oversamples comments with API scores between 0.5 and 0.8 where human disagreement is highest.

## Evidence (verbatim from paper)

> Coding achieved an interrater agreement Cohen’s kappa κ>0.9 for all three variants, indicating strong agreement. In the event that a rater ascribed multiple themes to a single open ended response, we required both raters to select the same set of themes to constitute agreement.

## Citation

```bibtex
@misc{kumar2021designing,
  title={Designing Toxic Content Classification for a Diversity of Perspectives},
  author={Kumar et al. (2021)},
  year={2021},
  note={arXiv:2106.04511}
}
```

- arXiv: 2106.04511