Hate Speech Ordinal Eval

---
name: hate-speech-ordinal-eval
description: Evaluates deep learning models' ability to predict continuous, interval-scaled hate speech scores from raw text comments. It benchmarks against existing APIs and transformer baselines using cross-validated error and correlation metrics. Use when the user wants to benchmark on Custom hate speech corpus (YouTube, Reddit, Twitter), or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2009.10277
  bibtex_key: kennedy2020constructing
  confidence: high
---

# hate-speech-ordinal-eval

> Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application — Kennedy et al. (2020) (arXiv:2009.10277, 2020)

## What this evaluates

Evaluates deep learning models' ability to predict continuous, interval-scaled hate speech scores from raw text comments. It benchmarks against existing APIs and transformer baselines using cross-validated error and correlation metrics.

## Datasets

- **Custom hate speech corpus (YouTube, Reddit, Twitter)** — total ?; splits: train (42000), val (-1); repo https://github.com/ck37/coral-ordinal

## Metrics

- `RMSE` **(primary)** — range: other
  - Root mean-squared error between predicted continuous hate scores and ground truth Rasch-calibrated scores.
- `MAE` — range: other
  - Mean absolute error between predicted and ground truth scores.
- `Corr` — range: [-1, 1]
  - Pearson linear correlation coefficient between predicted and ground truth scores.

## Input / output format

**Input**: Raw text of a user comment.

**Output**: A single continuous float representing the predicted hate speech score.

## Scoring recipe

```python
preds = model.predict(comments)
rmse = np.sqrt(np.mean((preds - gold) ** 2))
mae = np.mean(np.abs(preds - gold))
corr = np.corrcoef(preds, gold)[0, 1]
return rmse, mae, corr
```

## Common pitfalls

- The training data distribution is intentionally skewed during collection and does not reflect population-level hate speech prevalence.
- Twitter comments may be pre-filtered by the platform or API, leading to artificially low scores compared to YouTube/Reddit.
- Baseline models (Jigsaw) require linear OLS calibration to map binary probabilities to the continuous scale.

## Evidence (verbatim from paper)

> Direct prediction of the continuous hate score has currently achieved the lowest root mean-squared error (RMSE), although our proposed multitask networks that are transformed via IRT achieved comparable performance and slightly lower mean absolute error with the benefit of explainability.

## Citation

```bibtex
@misc{kennedy2020constructing,
  title={Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application},
  author={Kennedy et al. (2020)},
  year={2020},
  note={arXiv:2009.10277}
}
```

- arXiv: 2009.10277