Mmr1 Eval

---
name: mmr1-eval
description: Evaluates multimodal mathematical and logical reasoning capabilities of vision-language models. It probes complex multi-step problem solving, visual reasoning, logical deduction, and chart-based understanding across five diverse benchmarks. Use when the user wants to benchmark on MathVerse, MathVista, MathVision, LogicVista, ChartQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.21268
  bibtex_key: leng2025mmr1
  confidence: high
---

# mmr1-eval

> MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources — Sicong Leng et al. (arXiv:2509.21268, 2025)

## What this evaluates

Evaluates multimodal mathematical and logical reasoning capabilities of vision-language models. It probes complex multi-step problem solving, visual reasoning, logical deduction, and chart-based understanding across five diverse benchmarks.

## Datasets

- **MathVerse** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **MathVision** — total ?; splits: test (-1)
- **LogicVista** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions per benchmark, averaged across MathVerse, MathVista, MathVision, LogicVista, and ChartQA.

## Input / output format

**Input**: Multimodal prompts containing an image and a mathematical or logical question.

**Output**: A step-by-step reasoning chain followed by the final answer enclosed in \box{}.

## Scoring recipe

```python
def score(predictions, golds, benchmark):
    if benchmark in ['MathVision', 'MMMU']:
        return official_protocol_score(predictions, golds)
    if is_math_question(benchmark):
        return math_verify_score(predictions, golds)
    else:
        return exact_match_or_semantic_score(predictions, golds)

# Final metric is the average accuracy across the 5 benchmarks.
```

## Common pitfalls

- Benchmarks have varying official protocols (e.g., MathVision, MMMU) that must be strictly followed rather than applying a single uniform parser.
- Math questions require specialized verifiers (Math-Verify/MathRuler) while non-math questions rely on exact matching, requiring careful routing.
- The \box{} delimiter must be correctly extracted; missing it causes false negatives even if the answer is present in the text.

## Evidence (verbatim from paper)

> We adopt a unified prompt across all evaluations, requiring models to enclose final answers in “\box{}” (full prompt in Appendix[D]). Inference is performed using vLLM for efficient generation. For benchmarks with official protocols (e.g., MathVision, MMMU), we strictly follow the original procedures. For others, mathematical questions are assessed with Math-Verify (Kydlíček, [2025]) and MathRuler (hiyouga, [2025]), while non-mathematical ones use exact matching. To ensure robustness, we further (1) select the most semantically similar option when multiple-choice answers do not exactly match any candidate, and (2) employ GPT-4o (OpenAI, [2024]) as an auxiliary judge for open-ended questions where exact matching or extraction fails.

## Citation

```bibtex
@misc{leng2025mmr1,
  title={MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources},
  author={Sicong Leng et al.},
  year={2025},
  note={arXiv:2509.21268}
}
```

- arXiv: 2509.21268