Offenseval 2020 Eval

---
name: offenseval-2020-eval
description: Binary classification of offensive versus non-offensive language in social media text across five languages (English, Arabic, Danish, Greek, Turkish). It probes multilingual transformer models' ability to detect hate speech and offensive content in both high-resource and low-resource settings. Use when the user wants to benchmark on SemEval-2020 Task 12 (Offenseval), or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.13609
  bibtex_key: tanase2020upb
  confidence: high
---

# offenseval-2020-eval

> UPB at SemEval-2020 Task 12: Multilingual Offensive Language Detection on Social Media by Fine-tuning a Variety of BERT-based Models — Tanase et al. (2020) (arXiv:2010.13609, 2020)

## What this evaluates

Binary classification of offensive versus non-offensive language in social media text across five languages (English, Arabic, Danish, Greek, Turkish). It probes multilingual transformer models' ability to detect hate speech and offensive content in both high-resource and low-resource settings.

## Datasets

- **SemEval-2020 Task 12 (Offenseval)** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `F1-score` **(primary)** — range: [0, 1] (reported as percent)
  - Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall). Computed per language on the validation set for model selection and on the hidden test set for leaderboard ranking.

## Input / output format

**Input**: Raw social media text (tweets) in one of five languages (English, Arabic, Danish, Greek, Turkish).

**Output**: Binary label indicating whether the text is offensive or non-offensive.

## Scoring recipe

```python
def compute_f1(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Using English-specific preprocessing (e.g., n-gram TFIDF) on non-English data severely degrades performance.
- Translating non-English tweets to English before classification yields poor results compared to using multilingual models.
- Model selection is based on validation set F1-score, which may not perfectly correlate with hidden test set performance.

## Evidence (verbatim from paper)

> The reported metrics are computed for the Offenseval 2020 language-specific validation sets as described in Section 4. For each language, the highest validation set F1-score is highlighted, meaning that the corresponding model was selected and employed for predicting the language-specific competition test data in our final submission.

## Citation

```bibtex
@misc{tanase2020upb,
  title={UPB at SemEval-2020 Task 12: Multilingual Offensive Language Detection on Social Media by Fine-tuning a Variety of BERT-based Models},
  author={Tanase et al. (2020)},
  year={2020},
  note={arXiv:2010.13609}
}
```

- arXiv: 2010.13609