Political Toxicity Annotation Eval

---
name: political-toxicity-annotation-eval
description: Evaluates the ability of LLMs and API-based classifiers to accurately annotate toxicity and incivility in political protest content against a human gold standard. It probes zero-shot classification performance, threshold sensitivity, and output reproducibility across different model sizes and temperatures. Use when the user wants to benchmark on Political protest content dataset, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.09741
  bibtex_key: gonzalezbustamante2024benchmarking
  confidence: medium
---

# political-toxicity-annotation-eval

> Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Data — González-Bustamante (2024) (arXiv:2409.09741, 2024)

## What this evaluates

Evaluates the ability of LLMs and API-based classifiers to accurately annotate toxicity and incivility in political protest content against a human gold standard. It probes zero-shot classification performance, threshold sensitivity, and output reproducibility across different model sizes and temperatures.

## Datasets

- **Political protest content dataset** — total ?; splits: test (-1)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used as the headline ranking metric in Table 1.
- `accuracy` — range: [0, 1]
  - Proportion of correct predictions compared to the human gold standard.
- `precision` — range: [0, 1]
  - Ability of the classifier to identify positive predicted values and avoid false negatives.
- `recall` — range: [0, 1]
  - Proportion of correct classifications among true-positive cases.

## Input / output format

**Input**: Raw text messages from political protest interactions.

**Output**: Binary classification labels (toxic/incivil vs. not) or probability scores (for Perspective API).

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    accuracy = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}
```

## Common pitfalls

- Perspective API performance is highly threshold-dependent; using the standard 0.70 cutoff drastically lowers F1-score compared to a 0.55 cutoff, changing its ranking from bottom to top.
- Model reproducibility varies significantly with temperature settings; GPT-4 shows extreme fluctuations at higher temperatures, while open-source models require minimum temperature for full reproducibility.
- Computing time and parameter count do not strictly correlate with performance; smaller models like Nous Hermes 2 and Mistral OpenOrca outperform larger ones in speed and sometimes accuracy.

## Evidence (verbatim from paper)

> The performance metrics are: (i) accuracy that reports the proportion of correct predictions of the particular classifier in comparison with the human gold standard; (ii) precision that shows the ability of the classifier to identify positive predicted values to identify false negatives; (iii) recall or sensitivity that shows the proportion of correct classifications among true-positive cases; and (iv) F1-score, a combination of precision and recall.

## Citation

```bibtex
@misc{gonzalezbustamante2024benchmarking,
  title={Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Data},
  author={González-Bustamante (2024)},
  year={2024},
  note={arXiv:2409.09741}
}
```

- arXiv: 2409.09741