Multimodal Math Reasoning Eval

---
name: multimodal-math-reasoning-eval
description: Evaluates the ability of multimodal large language models to solve mathematical problems that require interpreting visual diagrams alongside textual prompts. It probes complex reasoning capabilities across diverse difficulty levels and languages (English and Chinese). Use when the user wants to benchmark on MathVista, MathVerse, MathVision, OlympiadBench, WeMath, MMK12-test, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.01106
  bibtex_key: gao2026divagrp
  confidence: high
---

# multimodal-math-reasoning-eval

> DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage — Gao et al. (2026) (arXiv:2603.01106, 2026)

## What this evaluates

Evaluates the ability of multimodal large language models to solve mathematical problems that require interpreting visual diagrams alongside textual prompts. It probes complex reasoning capabilities across diverse difficulty levels and languages (English and Chinese).

## Datasets

- **MathVista** — total ?; splits: test (-1)
- **MathVerse** — total ?; splits: test (-1)
- **MathVision** — total ?; splits: test (-1)
- **OlympiadBench** — total ?; splits: test (-1)
- **WeMath** — total ?; splits: test (-1)
- **MMK12-test** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered instances, calculated as the number of exact matches between the model's predicted answer and the ground truth divided by the total number of test instances.

## Input / output format

**Input**: Multimodal mathematical problems consisting of an image (diagram/figure) and a textual question.

**Output**: Textual reasoning steps followed by a final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- The benchmarks are multimodal, so evaluating text-only models or ignoring image perturbations/variants leads to unfair comparisons.
- The paper uses a subset (5,000 instances) for ablation studies on MMK12, not the full test set, which may affect generalization claims.
- Accuracy is reported as a simple average across benchmarks, which might mask performance disparities on specific domains or languages.

## Evidence (verbatim from paper)

> As reported in Table 2, removing any single component consistently decreases performance, with the full DIVA-GRPO model achieving the highest accuracy across all benchmarks. These results indicate that all components contribute complementary gains and none can be omitted without performance degradation, highlighting the necessity of the complete model design.

## Citation

```bibtex
@misc{gao2026divagrp,
  title={DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2603.01106}
}
```

- arXiv: 2603.01106