Gpro Eval

---
name: gpro-eval
description: Evaluates large vision-language models on complex mathematical and visual reasoning tasks, measuring both correctness and computational efficiency. It specifically probes the model's ability to avoid excessive chain-of-thought generation (overthinking) by dynamically routing computation between fast perception, slow perception, and slow reasoning paths. Use when the user wants to benchmark on MathVision, MathVerse, MathVista, DynaMath, MM-Vet, or asks about evaluating this task. Reports accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.04442
  bibtex_key: diao2026gpro
  confidence: high
---

# gpro-eval

> Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization — Diao et al. (2026) (arXiv:2601.04442, 2026)

## What this evaluates

Evaluates large vision-language models on complex mathematical and visual reasoning tasks, measuring both correctness and computational efficiency. It specifically probes the model's ability to avoid excessive chain-of-thought generation (overthinking) by dynamically routing computation between fast perception, slow perception, and slow reasoning paths.

## Datasets

- **MathVision** — total ?; splits: test (-1)
- **MathVerse** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **DynaMath** — total ?; splits: test (-1)
- **MM-Vet** — total ?; splits: test (-1)

## Metrics

- `accuracy (%)` **(primary)** — range: percent
  - Percentage of questions answered correctly out of the total number of questions in the benchmark.
- `average response length (tokens)` — range: other
  - Mean number of generated tokens per question across the benchmark test set.

## Input / output format

**Input**: Multimodal input consisting of one or more images paired with a text prompt describing a mathematical, geometric, or spatial reasoning problem.

**Output**: Natural language text containing step-by-step reasoning (optional) and a final answer.

## Scoring recipe

```python
def compute_metrics(predictions, gold_answers, token_counts):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if pred == gold)
    accuracy = (correct / len(gold_answers)) * 100
    avg_tokens = sum(token_counts) / len(token_counts)
    return {'accuracy (%)': accuracy, 'average response length (tokens)': avg_tokens}
```

## Common pitfalls

- Assuming longer chain-of-thought generation always improves accuracy, whereas the paper shows it often introduces noise and hallucination.
- Failing to distinguish between perceptual failures (misreading visual elements) and reasoning failures (logical errors), which require different mitigation strategies.
- Evaluating efficiency solely on wall-clock time without accounting for token generation length, which directly correlates with computational cost in autoregressive models.

## Evidence (verbatim from paper)

> Table[2] presents a comprehensive comparison across five multimodal reasoning benchmarks. We report both accuracy (%) and average response length (tokens) to facilitate a dual analysis of performance and computational efficiency.

## Citation

```bibtex
@misc{diao2026gpro,
  title={Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization},
  author={Diao et al. (2026)},
  year={2026},
  note={arXiv:2601.04442}
}
```

- arXiv: 2601.04442