M2rag Multimodal Eval

---
name: m2rag-multimodal-eval
description: Evaluates multimodal retrieval-augmented generation systems across open-domain question answering, image captioning, and fact verification. It measures how effectively a system selects and utilizes retrieved multimodal evidence to improve generation quality and factual accuracy. Use when the user wants to benchmark on M2RAG, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.24564
  bibtex_key: wang2026megrag
  confidence: high
---

# m2rag-multimodal-eval

> MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAG — Wang et al. (2026) (arXiv:2604.24564, 2026)

## What this evaluates

Evaluates multimodal retrieval-augmented generation systems across open-domain question answering, image captioning, and fact verification. It measures how effectively a system selects and utilizes retrieved multimodal evidence to improve generation quality and factual accuracy.

## Datasets

- **M2RAG** — total ?; splits: test (-1)

## Metrics

- `CIDEr` **(primary)** — range: [0, 1]
  - Consensus-based Image Description Evaluation. Measures n-gram overlap between generated and reference captions, weighted by consensus among multiple references.
- `BLEU` — range: [0, 1]
  - Bilingual Evaluation Understudy. Measures precision of n-gram matches between generated and reference text.
- `ROUGE` — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation. Measures recall of n-gram overlap between generated and reference text.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly verified claims or answered questions.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for fact verification tasks.

## Input / output format

**Input**: Multimodal query (text and/or image) with a candidate set of retrieved multimodal documents provided as context.

**Output**: Natural language generation (answer, caption, or verification statement).

## Scoring recipe

```python
def score(predictions, references):
    bleu = corpus_bleu(references, predictions)
    rouge = corpus_rouge(references, predictions)
    cider = corpus_cider(references, predictions)
    acc = mean([p == r for p, r in zip(predictions, references)])
    f1 = f1_score(references, predictions, average='macro')
    return {'BLEU': bleu, 'ROUGE': rouge, 'CIDEr': cider, 'Accuracy': acc, 'F1': f1}
```

## Common pitfalls

- Threshold sensitivity for MEG labeling (τ) significantly impacts training data balance and downstream performance.
- Loss weight (α) between CE and RankNet losses requires careful tuning; extreme values yield sub-optimal results.
- Teacher model choice can introduce bias if not properly distilled, though MEG-RAG shows robustness.

## Evidence (verbatim from paper)

> Following the M2RAG benchmark, we report BLEU, ROUGE, and CIDEr for MMQA and Image Captioning, and Accuracy and F1 for Fact Verification.

## Citation

```bibtex
@misc{wang2026megrag,
  title={MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAG},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.24564}
}
```

- arXiv: 2604.24564