Olid Eval

---
name: olid-eval
description: Evaluates models on a three-level hierarchical schema for offensive language in social media, probing the ability to detect offensiveness, categorize offense type, and identify the target of the offensive content. Use when the user wants to benchmark on Offensive Language Identification Dataset (OLID), or asks about evaluating this task. Reports macro-averaged F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1902.09666
  bibtex_key: zampieri2019predicting
  confidence: high
---

# olid-eval

> Predicting the Type and Target of Offensive Posts in Social Media — Zampieri et al. (2019) (arXiv:1902.09666, 2019)

## What this evaluates

Evaluates models on a three-level hierarchical schema for offensive language in social media, probing the ability to detect offensiveness, categorize offense type, and identify the target of the offensive content.

## Datasets

- **Offensive Language Identification Dataset (OLID)** — total 10000; splits: train (-1), test (-1)

## Metrics

- `macro-averaged F1-score` **(primary)** — range: [0, 1]
  - The unweighted mean of the F1-score computed independently for each class, then averaged across all classes. This metric treats all classes equally regardless of their frequency in the dataset.

## Input / output format

**Input**: Raw text of a social media post (tweet).

**Output**: Predicted class label(s) corresponding to the hierarchical level: Level A (OFF/NOT), Level B (TIN/UNT), or Level C (GRP/IND/OTH).

## Scoring recipe

```python
def macro_f1(y_true, y_pred):
    classes = set(y_true)
    f1s = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1s.append(f1)
    return sum(f1s) / len(f1s)
```

## Common pitfalls

- The dataset is highly imbalanced across classes; using accuracy instead of macro-F1 will misrepresent model performance.
- The 'OTH' (others) target class has very few training instances (395) and is semantically heterogeneous, often causing models to achieve 0 performance on it.
- Tasks are evaluated independently in the reported experiments rather than as a strict hierarchical cascade, which may overestimate real-world pipeline performance.

## Evidence (verbatim from paper)

> As the label distribution is highly imbalanced (see Table[3](#S3.T3 "Table 3 ‣ 3 Data Collection ‣ Predicting the Type and Target of Offensive Posts in Social Media")), we evaluate and we compare the performance of the different models using macro-averaged F1-score. We further report per-class Precision (P), Recall (R), and F1-score (F1), and weighted average.

## Citation

```bibtex
@misc{zampieri2019predicting,
  title={Predicting the Type and Target of Offensive Posts in Social Media},
  author={Zampieri et al. (2019)},
  year={2019},
  note={arXiv:1902.09666}
}
```

- arXiv: 1902.09666