Ethos Hate Speech Eval

---
name: ethos-hate-speech-eval
description: Evaluates the ability of NLP models to detect and classify hate speech in social media comments. It probes both binary hate/non-hate classification and multi-label categorization across specific demographic/identity-based hate categories. The benchmark emphasizes handling overlapping labels and class imbalance typical of real-world user-generated text. Use when the user wants to benchmark on ETHOS, or asks about evaluating this task. Reports F1-score (macro).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2006.08328
  bibtex_key: mollas2020ethos
  confidence: high
---

# ethos-hate-speech-eval

> ETHOS: an Online Hate Speech Detection Dataset — Mollas et al. (2020) (arXiv:2006.08328, 2020)

## What this evaluates

Evaluates the ability of NLP models to detect and classify hate speech in social media comments. It probes both binary hate/non-hate classification and multi-label categorization across specific demographic/identity-based hate categories. The benchmark emphasizes handling overlapping labels and class imbalance typical of real-world user-generated text.

## Datasets

- **ETHOS** — total ?; splits: train (-1), test (-1); repo https://github.com/intelligence-csd-auth-gr/Ethos-Hate-Speech-Dataset.git

## Metrics

- `F1-score (macro)` **(primary)** — range: percent
  - Macro-averaged F1 score computed across all hate speech categories, treating each label independently and averaging their F1 scores.
- `Accuracy` — range: percent
  - Standard classification accuracy calculated as the proportion of correctly predicted instances out of the total.
- `Hamming Loss` — range: [0, 1]
  - Symmetric difference between ground truth and predicted labels, normalized by the total number of labels.
- `Subset Accuracy` — range: percent
  - Exact match ratio where the predicted label set must perfectly match the ground truth label set for an instance to be counted as correct.

## Input / output format

**Input**: Raw social media comments (YouTube/Reddit) preprocessed via lowercasing, contraction expansion, punctuation removal, and stemming/lemmatization (stemming/lemmatization explicitly skipped for Text-to-Sequence models).

**Output**: Binary label {0, 1} per category, derived by thresholding initial [0,1] discrete values at ≥0.5. Multi-label scope outputs a binary vector per instance.

## Scoring recipe

```python
# Binarize initial [0,1] values using 0.5 threshold
pred_bin = [1 if p >= 0.5 else 0 for p in pred]
gold_bin = [1 if g >= 0.5 else 0 for g in gold]

# Binary scope metrics
acc = accuracy_score(gold_bin, pred_bin)
f1_macro = f1_score(gold_bin, pred_bin, average='macro')

# Multi-label scope metrics
hamming = hamming_loss(gold_bin, pred_bin)
subset_acc = subset_accuracy(gold_bin, pred_bin)
precision_macro = precision_score(gold_bin, pred_bin, average='macro')
recall_macro = recall_score(gold_bin, pred_bin, average='macro')
```

## Common pitfalls

- Dataset labels are initially discrete in [0,1] and must be binarized at a 0.5 threshold before evaluation; skipping this step yields incorrect scores.
- Evaluation relies on nested cross-validation for traditional ML and 10-fold CV for neural networks rather than a fixed held-out test set, which affects how results should be aggregated and compared.
- Stemming and lemmatization are explicitly skipped for Text-to-Sequence models (e.g., BERT, DistilBERT); applying them violates the stated protocol.

## Evidence (verbatim from paper)

> We chose accuracy and precision, recall and $F_{1}$-score with macro indication, and the confusion matrix as metrics. ... In the evaluation of MLL systems, a very common measure is the Hamming loss (symmetric difference between the ground truth labels and the predicted ones). Furthermore, subset accuracy (symmetric similarity), as well as precision, recall and $F_{1}$-score, are contained here (instance-based metrics).

## Citation

```bibtex
@misc{mollas2020ethos,
  title={ETHOS: an Online Hate Speech Detection Dataset},
  author={Mollas et al. (2020)},
  year={2020},
  note={arXiv:2006.08328}
}
```

- arXiv: 2006.08328