M4 Rag Eval

---
name: m4-rag-eval
description: Evaluates vision-language models on multilingual, multicultural, and multimodal retrieval-augmented generation tasks. It measures how different retrieval strategies, language alignment, and model scale impact accuracy on culturally diverse image-question pairs. Use when the user wants to benchmark on CVQA, WorldCuisines, or asks about evaluating this task. Reports macro-averaged accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.05959
  bibtex_key: anugraha2025m4rag
  confidence: high
---

# m4-rag-eval

> M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG — Anugraha et al. (2025) (arXiv:2512.05959, 2025)

## What this evaluates

Evaluates vision-language models on multilingual, multicultural, and multimodal retrieval-augmented generation tasks. It measures how different retrieval strategies, language alignment, and model scale impact accuracy on culturally diverse image-question pairs.

## Datasets

- **CVQA** — total ?; splits: test (-1)
- **WorldCuisines** — total ?; splits: test (-1)

## Metrics

- `macro-averaged accuracy` **(primary)** — range: [0, 1]
  - Average of per-class accuracies across all datasets, computed by comparing model-generated multiple-choice answers against ground-truth labels.

## Input / output format

**Input**: Image (I), question (q), and optional retrieved context passages (top-k=5). Prompts are provided in English or translated target languages.

**Output**: Multiple-choice answer selection from the provided options.

## Scoring recipe

```python
class_correct = defaultdict(int)
class_total = defaultdict(int)
for pred, gold in zip(predictions, gold_labels):
    class_total[gold] += 1
    if pred == gold:
        class_correct[gold] += 1
class_acc = [class_correct[c] / class_total[c] for c in class_total]
return sum(class_acc) / len(class_acc)
```

## Common pitfalls

- Confusing the six retrieval configurations (Baseline, Ground-Truth, Oracle-Query RAG, Caption+Question RAG, Multimodal RAG with mmE5/B3) as a single setting; each must be evaluated separately.
- Assuming VLM-as-a-judge replaces the primary metric; it is only used for annotation/reasoning evaluation, while macro-averaged accuracy is the headline metric for multiple-choice answers.
- Overlooking the cross-lingual alignment variable; performance is measured across language mismatches between prompts, context, and model capabilities.

## Evidence (verbatim from paper)

> For VLMs generations, we use macro-averaged accuracy for all datasets by comparing the multiple choice answer. For annotations we use VLM-as-a-judge using reasoning rubric based since it improves reasoning and more interpretable [3, 4, 19].

## Citation

```bibtex
@misc{anugraha2025m4rag,
  title={M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG},
  author={Anugraha et al. (2025)},
  year={2025},
  note={arXiv:2512.05959}
}
```

- arXiv: 2512.05959