Emma Multimodal Reasoning Eval

---
name: emma-multimodal-reasoning-eval
description: This benchmark evaluates multimodal large language models on their ability to perform integrated visual-textual reasoning across mathematics, physics, chemistry, and coding. It probes capabilities such as fine-grained spatial simulation, multi-hop visual inference, and cross-modal problem solving under both direct and chain-of-thought prompting conditions. Use when the user wants to benchmark on EMMA-mini, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.05444
  bibtex_key: hao2025emma
  confidence: high
---

# emma-multimodal-reasoning-eval

> Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark — Hao et al. (2025) (arXiv:2501.05444, 2025)

## What this evaluates

This benchmark evaluates multimodal large language models on their ability to perform integrated visual-textual reasoning across mathematics, physics, chemistry, and coding. It probes capabilities such as fine-grained spatial simulation, multi-hop visual inference, and cross-modal problem solving under both direct and chain-of-thought prompting conditions.

## Datasets

- **EMMA-mini** — total 400; splits: test (400)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of questions where the model's final predicted answer exactly matches the ground truth answer. Calculated as (number of correct predictions / total number of questions) * 100.
- `Pass@N` — range: percent
  - Upper-bound accuracy measuring whether at least one of N generated attempts contains the correct answer. Calculated as (number of questions with at least one correct response among N attempts / total questions) * 100.

## Input / output format

**Input**: Multimodal question instances containing an image and a text prompt, asking for solutions in math, physics, chemistry, or coding. Models receive either a direct instruction to output the answer or a Chain-of-Thought prompt instructing them to 'think step-by-step'.

**Output**: A final answer string (and optionally a step-by-step reasoning trace if CoT prompting is used).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if normalize(pred) == normalize(gold))
    return (correct / len(gold_answers)) * 100

def compute_pass_at_n(predictions_per_question, gold_answers):
    correct_count = 0
    for preds, gold in zip(predictions_per_question, gold_answers):
        if any(normalize(p) == normalize(gold) for p in preds):
            correct_count += 1
    return (correct_count / len(gold_answers)) * 100
```

## Common pitfalls

- Confusing the full EMMA benchmark with EMMA-mini, a randomly sampled 400-question subset used for detailed analysis and human baselines.
- Assuming Chain-of-Thought (CoT) prompting universally improves performance; the paper shows it significantly degrades accuracy for open-source models.
- Treating test-time scaling results (N=1 to 16) as standard evaluation conditions rather than ablation studies to probe reasoning upper bounds.

## Evidence (verbatim from paper)

> On EMMA-mini, the best-performing model, o1, achieves an accuracy of 45.75%, trailing human experts by 32%. At the lower end, LLaVA-OneVision-72B scores only 25.25%, barely surpassing random choice by 2.5%.

## Citation

```bibtex
@misc{hao2025emma,
  title={Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark},
  author={Hao et al. (2025)},
  year={2025},
  note={arXiv:2501.05444}
}
```

- arXiv: 2501.05444