Tweettopic Eval

---
name: tweettopic-eval
description: Evaluates language models on classifying tweets into predefined topics, testing both single-label and multi-label classification capabilities. It probes robustness to social media noise, short-form content, and topic overlap in real-world settings. Use when the user wants to benchmark on TweetTopic, or asks about evaluating this task. Reports Macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.09824
  bibtex_key: [REDACTED]
  confidence: high
---

# tweettopic-eval

> Twitter Topic Classification — Antypas et al. (2022) (arXiv:2209.09824, 2022)

## What this evaluates

Evaluates language models on classifying tweets into predefined topics, testing both single-label and multi-label classification capabilities. It probes robustness to social media noise, short-form content, and topic overlap in real-world settings.

## Datasets

- **TweetTopic** — total 11267; splits: train (-1), test (-1)

## Metrics

- `Macro Precision` — range: percent
  - Calculated per class as TP/(TP+FP), then averaged across all classes.
- `Macro Recall` — range: percent
  - Calculated per class as TP/(TP+FN), then averaged across all classes.
- `Macro F1` **(primary)** — range: percent
  - Harmonic mean of macro Precision and macro Recall, averaged across all classes.
- `Accuracy` — range: percent
  - Proportion of correctly classified tweets out of the total.
- `Jaccard Index` — range: percent
  - For each tweet, size of intersection over union of predicted and true label sets. Final metric is the average over all tweets.

## Input / output format

**Input**: Raw tweet text.

**Output**: A single topic label (single-label) or a set of topic labels (multi-label).

## Scoring recipe

```python
def compute_macro_f1(preds, golds, classes):
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return sum(f1s) / len(f1s)

def compute_jaccard(preds, golds):
    scores = []
    for p, g in zip(preds, golds):
        inter = len(set(p) & set(g))
        union = len(set(p) | set(g))
        scores.append(inter / union if union > 0 else 0)
    return sum(scores) / len(scores)
```

## Common pitfalls

- TimeLM-21 was trained on a corpus overlapping with the test set period, giving it an unfair advantage and inflating its reported performance.
- Multi-label classification is significantly harder than single-label due to label overlap and social media noise, often resulting in much lower F1 scores.
- Macro averaging can mask poor performance on minority topics if not analyzed per-class.

## Evidence (verbatim from paper)

> For both settings macro average Precision, Recall and F1, as well as Accuracy, are used to evaluate the models tested. As an alternative metric for the multi-label setting, Jaccard Index (JI) is also utilized, as it can offer useful insights about the models performances (Pereira et al., 2018; Tsoumakas et al., 2009). More specifically, the index is calculated for each tweet individually and the final metric is computed as the average over all entries.

## Citation

```bibtex
@misc{antypas2022twitter,
  title={Twitter Topic Classification},
  author={Antypas et al. (2022)},
  year={2022},
  note={arXiv:2209.09824}
}
```

- arXiv: 2209.09824