Memelens Multitask Eval

---
name: memelens-multitask-eval
description: Evaluates multimodal vision-language models on understanding memes across multiple languages and semantic categories. It probes cross-modal reasoning, cross-lingual transfer, and the ability to generalize across diverse tasks like harm detection, misinformation, and humor/sarcasm. Use when the user wants to benchmark on MemeLens Unified Benchmark, or asks about evaluating this task. Reports Macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.12539
  bibtex_key: shahroor2026memelens
  confidence: high
---

# memelens-multitask-eval

> MemeLens: Multilingual Multitask VLMs for Memes — Shahroor et al. (2026) (arXiv:2601.12539, 2026)

## What this evaluates

Evaluates multimodal vision-language models on understanding memes across multiple languages and semantic categories. It probes cross-modal reasoning, cross-lingual transfer, and the ability to generalize across diverse tasks like harm detection, misinformation, and humor/sarcasm.

## Datasets

- **MemeLens Unified Benchmark** — total ?; splits: test (-1)

## Metrics

- `Accuracy` — range: [0, 1]
  - Proportion of correctly predicted instances out of the total number of instances.
- `Macro-F1` **(primary)** — range: [0, 1]
  - Unweighted mean of the F1 scores for each class. Calculated as the average of recall and precision per class, then averaged across all classes to handle class imbalance.
- `Weighted-F1` — range: [0, 1]
  - Mean of the F1 scores for each class, weighted by the number of true instances for each class (support).

## Input / output format

**Input**: Multimodal meme instance consisting of an image and its embedded text.

**Output**: Discrete class label from the shared 20-task taxonomy. Explanation-augmented training also requires a natural language rationale.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    accuracy = (y_true == y_pred).mean()
    macro_f1 = f1_score(y_true, y_pred, average='macro')
    weighted_f1 = f1_score(y_true, y_pred, average='weighted')
    return accuracy, macro_f1, weighted_f1
```

## Common pitfalls

- Source datasets use heterogeneous evaluation protocols and label granularities, requiring careful metric alignment (Macro-F1 vs Weighted-F1 vs POS-F1) for fair comparison.
- The unified benchmark filters out samples without embedded text, which alters the data distribution and reduces direct comparability with prior work on unfiltered meme datasets.
- Single-dataset fine-tuning causes over-specialization to specific annotation conventions, so performance on one benchmark does not indicate cross-task generalization.

## Evidence (verbatim from paper)

> We report Accuracy, Macro-F1 (the primary metric due to class imbalance), and Weighted-F1 for this evaluation and analysis.

## Citation

```bibtex
@misc{shahroor2026memelens,
  title={MemeLens: Multilingual Multitask VLMs for Memes},
  author={Shahroor et al. (2026)},
  year={2026},
  note={arXiv:2601.12539}
}
```

- arXiv: 2601.12539