Multimodal Reasoning Eval

---
name: multimodal-reasoning-eval
description: This evaluation probes the ability of multimodal large language models to perform complex reasoning across diverse domains (mathematics, science, diagram comprehension, and creative tasks) by requiring them to explicitly ground their reasoning in visual and textual evidence before producing a final answer. Use when the user wants to benchmark on MMMU, MathVista, AI2D, EMMA, Creation-MMBench, Creation-MMBench-TO, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.24776
  bibtex_key: ding2025vtpereptionr1
  confidence: high
---

# multimodal-reasoning-eval

> VTPerception-R1: Enhancing Multimodal Reasoning via Explicit Visual and Textual Perceptual Grounding — Ding et al. (2025) (arXiv:2509.24776, 2025)

## What this evaluates

This evaluation probes the ability of multimodal large language models to perform complex reasoning across diverse domains (mathematics, science, diagram comprehension, and creative tasks) by requiring them to explicitly ground their reasoning in visual and textual evidence before producing a final answer.

## Datasets

- **MMMU** — total 11500; splits: test (-1)
- **MathVista** — total 6141; splits: test (-1)
- **AI2D** — total ?; splits: test (-1)
- **EMMA** — total ?; splits: test (-1)
- **Creation-MMBench** — total 765; splits: test (-1)
- **Creation-MMBench-TO** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard benchmark accuracy calculated as the percentage of correctly answered questions out of the total number of questions in each benchmark.

## Input / output format

**Input**: Image-text pairs containing a visual input and a question/prompt requiring reasoning.

**Output**: Structured text containing three distinct sections: <description> (task-relevant perceptual evidence), <think> (reasoning chain), and <answer> (final response).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        # Extract answer from structured output
        ans = pred.split('<answer>')[-1].strip().rstrip('</answer>')
        if ans == gold_ans:
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Confusing the training datasets (LLaVA-CoT, Vision-SR1, MMK12, etc.) with the evaluation benchmarks.
- Failing to parse the structured <description>, <think>, <answer> format correctly when extracting the final answer for scoring.
- Applying uniform scoring across benchmarks without accounting for dataset-specific formats (e.g., multiple-choice vs. open-ended).

## Evidence (verbatim from paper)

> The reward function combined answer accuracy, format compliance, key visual/textual information, n-gram penalty, and consistency, with tuned weights.

## Citation

```bibtex
@misc{ding2025vtpereptionr1,
  title={VTPerception-R1: Enhancing Multimodal Reasoning via Explicit Visual and Textual Perceptual Grounding},
  author={Ding et al. (2025)},
  year={2025},
  note={arXiv:2509.24776}
}
```

- arXiv: 2509.24776