Mmfinereason Eval

---
name: mmfinereason-eval
description: Evaluates multimodal reasoning capabilities across STEM, puzzles, general VQA, and document understanding domains. Probes how well vision-language models perform on complex visual reasoning tasks under strict greedy decoding and high-resolution inference settings. Use when the user wants to benchmark on MMMU_val, MathVista_mini, MathVision_test, MathVerse_mini, Dynamath, LogicVista, VisuLogic, ScienceQA, RealWorldQA, MMBench-EN, MMStar_test, AI2D_test, CharXiv_reas, CharXiv_desc, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.21821
  bibtex_key: lin2026mmfinereason
  confidence: high
---

# mmfinereason-eval

> MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods — Honglin Lin et al. (2026) (arXiv:2601.21821, 2026)

## What this evaluates

Evaluates multimodal reasoning capabilities across STEM, puzzles, general VQA, and document understanding domains. Probes how well vision-language models perform on complex visual reasoning tasks under strict greedy decoding and high-resolution inference settings.

## Datasets

- **MMMU_val** — total ?; splits: val (-1)
- **MathVista_mini** — total ?; splits: mini (-1)
- **MathVision_test** — total ?; splits: test (-1)
- **MathVerse_mini** — total ?; splits: mini (-1)
- **Dynamath** — total ?; splits: test (-1)
- **LogicVista** — total ?; splits: test (-1)
- **VisuLogic** — total ?; splits: test (-1)
- **ScienceQA** — total ?; splits: test (-1)
- **RealWorldQA** — total ?; splits: test (-1)
- **MMBench-EN** — total ?; splits: EN (-1)
- **MMStar_test** — total ?; splits: test (-1)
- **AI2D_test** — total ?; splits: test (-1)
- **CharXiv_reas** — total ?; splits: reas (-1)
- **CharXiv_desc** — total ?; splits: desc (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions per benchmark. Calculated as (number of correct predictions / total number of samples) × 100.

## Input / output format

**Input**: Image-text prompts where images are resized to 2048×2048 during inference. Models are evaluated in 'thinking mode' with greedy decoding (Temperature = 0).

**Output**: Textual reasoning chains followed by final answers, generated via greedy decoding.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred.strip() == gold.strip())
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Using temperature > 0 during evaluation; the protocol strictly requires greedy decoding (Temperature = 0) to assess reasoning reliability.
- Evaluating at low image resolution; the protocol mandates increasing maximum resolution to 2048×2048 during inference for fair comparison.
- Comparing models not in 'thinking mode'; all baselines and proposed models must be evaluated in thinking mode to ensure consistency.

## Evidence (verbatim from paper)

> MMFineReason demonstrates superior data efficiency, achieving higher accuracy with significantly fewer samples and smaller model parameters compared to MMR1 and HoneyBee.

## Citation

```bibtex
@misc{lin2026mmfinereason,
  title={MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods},
  author={Honglin Lin et al. (2026)},
  year={2026},
  note={arXiv:2601.21821}
}
```

- arXiv: 2601.21821