Mmmu Eval

---
name: mmmu-eval
description: Evaluate a multimodal model (LMM) on MMMU — 11.5K college-level questions across 6 disciplines, 30 subjects, 30 image types (charts, MRI, music sheets, chemical structures...). Use when the user wants to benchmark a vision-language model's expert-level reasoning, mentions MMMU / MMMU-Pro, or asks "is my LMM at expert human level?". Reports micro-averaged accuracy.
metadata:
  source_paper: arXiv:2311.16502
  source_repo: https://github.com/MMMU-Benchmark/MMMU
  task_type: multimodal-qa
  metrics: [accuracy_micro]
  splits: [dev (150), validation (900), test (10500)]
---

# mmmu-eval — Multi-discipline multimodal expert-level QA

Source: Yue et al., "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI" (CVPR 2024, arXiv:2311.16502).

## What this benchmark tests

College-level multimodal reasoning. 11,550 manually curated questions from textbooks, exams, quizzes across 6 disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Sci, Tech & Engineering), 30 subjects, 183 subfields. 30 image types — way beyond natural photos: chemical structures, geometric shapes, music sheets, MRI scans, circuit diagrams, technical schematics.

## When to invoke this skill

- User wants to test an LMM on "expert-level" / "college-level" / "domain-knowledge" multimodal tasks
- User mentions `MMMU`, `MMMU-Pro`, `MMMU-Plus`, "expert AGI"
- User asks "how does my VLM stack up on technical disciplines?" (vs general visual QA like VQAv2)

## Dataset structure

Each example has:
```json
{
  "id": "Validation_1",
  "question": "Based on the chart, which company had the highest revenue in Q3?",
  "options": "['A) Acme', 'B) Globex', 'C) Initech', 'D) Umbrella']",
  "answer": "B",
  "question_type": "multiple-choice",        // or "open"
  "image_1": <PIL.Image>,                     // up to image_7 — interleaved
  "subject": "Finance",
  "subfield": "Corporate Finance",
  "img_type": "['Charts']",
  "topic_difficulty": "Medium"
}
```

Splits (HF `MMMU/MMMU`, 30 subject configs):
- `dev`: 150 (5/subject) — for prompt-tuning, public answers
- `validation`: 900 (30/subject) — public answers, **report this for any model**
- `test`: 10,500 — **answers held out**, must submit to leaderboard at https://mmmu-benchmark.github.io

## Evaluation protocol (verbatim from §4)

1. **Setting**: zero-shot only; no fine-tuning on MMMU train data (there is none anyway).
2. **Prompt**: each model uses its own preferred prompt; for unfamiliar models, prompt-engineer on `dev` and reuse.
3. **Question types**:
   - Multi-choice (~94% of items): model outputs free text, scorer regexes for the option letter.
   - Open: model outputs free text, scorer regexes for key phrases / numbers.
4. **Scoring (rule-based, not LLM-judge)**:
   - Multi-choice: extract first letter from response; if no valid letter, assign **random** answer (paper's tie-breaker).
   - Open: extract numeric / phrase via regex; if invalid, mark **incorrect**.
5. **Metric**: `micro-averaged accuracy` over all questions in the split.

## Reference scoring (paper's official extractor, abridged)

```python
import re
def extract_mc_answer(response: str, options: list[str]) -> str:
    # Try patterns "A)", "(A)", "Answer: A", final letter A/B/C/D
    for pat in [r"\b([A-Z])\)", r"\(([A-Z])\)", r"answer is\s*([A-Z])",
                r"\b([A-Z])\b\s*$"]:
        m = re.search(pat, response, re.IGNORECASE)
        if m and m.group(1).upper() in [chr(ord('A')+i) for i in range(len(options))]:
            return m.group(1).upper()
    import random; return random.choice([chr(ord('A')+i) for i in range(len(options))])

def micro_acc(predictions: list[dict], gold: list[dict]) -> float:
    correct = sum(p["pred"] == g["answer"] for p, g in zip(predictions, gold))
    return correct / len(gold)
```

Use the official scorer in the repo (`mmmu/eval/eval_utils.py`) when reporting numbers — it has many corner cases beyond the sketch above.

## Output format the agent must produce

For each instance: `{"id": "Validation_1", "response": "<free text>", "pred": "B"}`.

When submitting to the test leaderboard, follow the JSON schema at https://github.com/MMMU-Benchmark/MMMU#leaderboard — micro-acc per discipline + overall.

## Don'ts

- Don't fine-tune on dev/validation and report the same split.
- Don't use an LLM judge on this — the official protocol is rule-based; LLM-judge introduces noise that breaks comparability with leaderboard numbers.
- Don't ignore image_2..image_7 — many items are interleaved (e.g. multiple subfigures); concatenating images is wrong.
- Don't conflate MMMU and MMMU-Pro (which is a harder, deduplicated subset with different scores).

## Citation

```bibtex
@inproceedings{yue2024mmmu,
  title={{MMMU}: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI},
  author={Yue, Xiang and Ni, Yuansheng and Zhang, Kai and others},
  booktitle={CVPR}, year={2024}
}
```