Facebook Hateful Memes Eval

---
name: facebook-hateful-memes-eval
description: This benchmark evaluates multimodal hate speech detection by classifying image-caption pairs (memes) as hateful or non-hateful. It probes a model's ability to align visual and textual cues while resisting spurious correlations, particularly under different prompt structures and data augmentation strategies. Use when the user wants to benchmark on Facebook Hateful Memes, or asks about evaluating this task. Reports weighted-F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.11808
  bibtex_key: singh2025labelsorinput
  confidence: high
---

# facebook-hateful-memes-eval

> Labels or Input? Rethinking Augmentation in Multimodal Hate Detection — Singh et al. (2025) (arXiv:2508.11808, 2025)

## What this evaluates

This benchmark evaluates multimodal hate speech detection by classifying image-caption pairs (memes) as hateful or non-hateful. It probes a model's ability to align visual and textual cues while resisting spurious correlations, particularly under different prompt structures and data augmentation strategies.

## Datasets

- **Facebook Hateful Memes** — total 16940; splits: train (12900), val (1040), test (3000)

## Metrics

- `accuracy` — range: percent
  - Number of correctly classified instances divided by the total number of instances.
- `weighted-F1 score` **(primary)** — range: percent
  - Harmonic mean of precision and recall, weighted by the number of true instances for each class (support).

## Input / output format

**Input**: Paired image and natural language caption representing a meme.

**Output**: Binary classification label: 1 (hateful) or 0 (non-hateful).

## Scoring recipe

```python
def compute_metrics(preds, gold):
    acc = sum(p == g for p, g in zip(preds, gold)) / len(gold)
    f1_scores = []
    for c in [0, 1]:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1 * sum(1 for g in gold if g == c))
    w_f1 = sum(f1_scores) / len(gold)
    return acc, w_f1
```

## Common pitfalls

- Using the original validation split for augmented experiments instead of the newly created 80:20 train-validation split.
- Relying solely on accuracy for imbalanced hate speech datasets without reporting weighted-F1, which masks poor minority-class performance.
- Mixing up binary and scaled label formats during prompt optimization, leading to inconsistent training objectives.

## Evidence (verbatim from paper)

> Our evaluation focuses on classification accuracy and weighted-F1 score, as standard in the hate classification benchmarks.

## Citation

```bibtex
@misc{singh2025labelsorinput,
  title={Labels or Input? Rethinking Augmentation in Multimodal Hate Detection},
  author={Singh et al. (2025)},
  year={2025},
  note={arXiv:2508.11808}
}
```

- arXiv: 2508.11808