Mrag Bench Eval

---
name: mrag-bench-eval
description: Evaluates large vision-language models' ability to leverage retrieved visual knowledge versus textual knowledge across perspective and transformative change scenarios. Probes robustness to noisy retrieved images and measures how effectively models utilize visually augmented information compared to human baselines. Use when the user wants to benchmark on MRAG-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.08182
  bibtex_key: hu2024mragbench
  confidence: high
---

# mrag-bench-eval

> MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models — Hu et al. (2024) (arXiv:2410.08182, 2024)

## What this evaluates

Evaluates large vision-language models' ability to leverage retrieved visual knowledge versus textual knowledge across perspective and transformative change scenarios. Probes robustness to noisy retrieved images and measures how effectively models utilize visually augmented information compared to human baselines.

## Datasets

- **MRAG-Bench** — total 1353; splits: test (1353)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard multiple-choice question accuracy. Calculated as the number of correctly answered questions divided by the total number of questions, expressed as a percentage.

## Input / output format

**Input**: Multi-image inputs paired with multiple-choice questions. Models are evaluated under three conditions: no additional knowledge, ground-truth image knowledge, and retrieved image knowledge from a multimodal retriever.

**Output**: A single multiple-choice answer. If pre-defined automatic extraction rules fail, GPT-3.5-turbo is used to extract the answer from the model's generation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if pred == gold)
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Automatic answer extraction rules may fail on some generations, requiring a fallback to GPT-3.5-turbo for reliable scoring.
- Open-source models often show performance degradation when provided with retrieved images compared to no RAG, due to difficulty filtering noisy or misleading visual examples.
- Performance varies significantly across the 9 fine-grained scenarios (e.g., perspective vs. transformative, incomplete/biological), so reporting only aggregate scores masks critical capability gaps.

## Evidence (verbatim from paper)

> We follow standard MCQA evaluation setup and employ accuracy score as our metric. We adopt default generation hyper-parameters selected by each model. Following *Lu et al. ([2024b])*, we employ GPT-3.5-turbo to extract the multiple choice answer in rare cases where our pre-defined automatic extraction rules failed.

## Citation

```bibtex
@misc{hu2024mragbench,
  title={MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models},
  author={Hu et al. (2024)},
  year={2024},
  note={arXiv:2410.08182}
}
```

- arXiv: 2410.08182