Hateful Meme Detection Eval

---
name: hateful-meme-detection-eval
description: Evaluates multimodal models' ability to detect hateful or offensive memes across multiple domains and under low-resource, out-of-distribution conditions. It probes robustness to distribution shifts, adversarial image perturbations, and the effectiveness of retrieval-augmented inference versus standard fine-tuning or in-context learning. Use when the user wants to benchmark on HatefulMemes, HarMeme, MAMI, Harm-P, MultiOFF, PrideMM, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.13061
  bibtex_key: mei2025robust
  confidence: high
---

# hateful-meme-detection-eval

> Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection — Mei et al. (2025) (arXiv:2502.13061, 2025)

## What this evaluates

Evaluates multimodal models' ability to detect hateful or offensive memes across multiple domains and under low-resource, out-of-distribution conditions. It probes robustness to distribution shifts, adversarial image perturbations, and the effectiveness of retrieval-augmented inference versus standard fine-tuning or in-context learning.

## Datasets

- **HatefulMemes** — total ?; splits: train (-1), test (-1)
- **HarMeme** — total ?; splits: train (-1), test (-1)
- **MAMI** — total ?; splits: train (-1), test (-1)
- **Harm-P** — total ?; splits: train (-1), test (-1)
- **MultiOFF** — total ?; splits: train (-1), test (-1)
- **PrideMM** — total ?; splits: train (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between hateful and non-hateful classes across all classification thresholds.
- `Accuracy` — range: [0, 1]
  - Ratio of correctly classified instances (both hateful and non-hateful) to the total number of instances.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Image and associated text/caption for each meme instance.

**Output**: Binary classification prediction (hateful vs. non-hateful) or class probability scores for AUC computation.

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import roc_auc_score, accuracy_score, f1_score

def compute_metrics(y_true, y_pred_proba, threshold=0.5):
    y_pred = (y_pred_proba >= threshold).astype(int)
    auc = roc_auc_score(y_true, y_pred_proba)
    acc = accuracy_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred)
    return {'AUC': auc, 'Accuracy': acc, 'F1': f1}
```

## Common pitfalls

- Using test set examples for few-shot or retrieval-based inference instead of the training split, which causes data leakage and inflates performance.
- Assuming few-shot in-context learning improves performance; the paper explicitly notes it is largely ineffective for meme classification and can degrade F1 on some datasets.
- Computing AUC for GPT-4o directly; the paper notes that token likelihoods are not accessible for this model, so AUC is reported as '-'.

## Evidence (verbatim from paper)

> RA-HMD improves upon RGCL with gains of over 4% in AUC and 3% in accuracy on HatefulMemes.

## Citation

```bibtex
@misc{mei2025robust,
  title={Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection},
  author={Mei et al. (2025)},
  year={2025},
  note={arXiv:2502.13061}
}
```

- arXiv: 2502.13061