Vl Rethinker Eval

---
name: vl-rethinker-eval
description: Evaluates the multimodal reasoning and self-reflection capabilities of vision-language models across math, multi-discipline, and real-world benchmarks. It probes whether models can correctly interpret visual-textual inputs and produce accurate final answers under greedy decoding. Use when the user wants to benchmark on MathVista, MathVerse, MathVision, MMMU-Pro, MMMU, EMMA, MegaBench, or asks about evaluating this task. Reports Pass@1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.08837
  bibtex_key: [REDACTED]
  confidence: high
---

# vl-rethinker-eval

> VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning — Wang et al. (2025) (arXiv:2504.08837, 2025)

## What this evaluates

Evaluates the multimodal reasoning and self-reflection capabilities of vision-language models across math, multi-discipline, and real-world benchmarks. It probes whether models can correctly interpret visual-textual inputs and produce accurate final answers under greedy decoding.

## Datasets

- **MathVista** — total ?; splits: testmini (-1)
- **MathVerse** — total ?; splits: testmini (-1)
- **MathVision** — total ?; splits: test (-1)
- **MMMU-Pro** — total ?; splits: overall (-1)
- **MMMU** — total ?; splits: val (-1), full (-1)
- **EMMA** — total ?; splits: full (-1)
- **MegaBench** — total ?; splits: core (-1)

## Metrics

- `Pass@1 accuracy` **(primary)** — range: percent
  - Calculated as the fraction of queries where the model's single greedy-decoded response exactly matches the ground truth answer.

## Input / output format

**Input**: Multimodal queries consisting of an image and a text prompt/question.

**Output**: A single final answer string generated via greedy decoding.

## Scoring recipe

```python
correct = 0
for query, gold in dataset:
    pred = model.generate(query, decoding='greedy')
    if normalize(pred) == normalize(gold):
        correct += 1
return (correct / len(dataset)) * 100
```

## Common pitfalls

- Using sampling or beam search instead of greedy decoding, which violates the specified evaluation protocol.
- Mixing up benchmark splits (e.g., using MathVista test instead of testmini, or MMMU full instead of val), as results are split-specific.
- Ignoring the multimodal nature of the input; the protocol strictly requires image-text pairs, not text-only queries.

## Evidence (verbatim from paper)

> For evaluation, we employ a diverse set of challenging multimodal benchmarks: MathVista, MathVerse, and MathVision. Multi-discipline understanding and reasoning: MMMU, MMMU-Pro, and EMMA. Large-scale long-tailed real-world tasks: MegaBench. This benchmark suite covers a wide range of complex multimodal reasoning challenges. We report the Pass@1 accuracy using greedy decoding.

## Citation

```bibtex
@misc{wang2025vlrethinker,
  title={VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2504.08837}
}
```

- arXiv: 2504.08837