Test Time Scaling Vlm Eval

---
name: test-time-scaling-vlm-eval
description: Evaluates the impact of test-time scaling (TTS) inference strategies on Vision-Language Models across multimodal reasoning and perception tasks. It measures how techniques like Chain-of-Thought, Best-of-N, Self-Consistency, and Self-Refinement improve or degrade performance on open-source versus closed-source models. Use when the user wants to benchmark on MathVista, MMMU, MMBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.11109
  bibtex_key: ahmadpour2025testtimescaling
  confidence: high
---

# test-time-scaling-vlm-eval

> Limits and Gains of Test-Time Scaling in Vision-Language Reasoning — Ahmadpour et al. (2025) (arXiv:2512.11109, 2025)

## What this evaluates

Evaluates the impact of test-time scaling (TTS) inference strategies on Vision-Language Models across multimodal reasoning and perception tasks. It measures how techniques like Chain-of-Thought, Best-of-N, Self-Consistency, and Self-Refinement improve or degrade performance on open-source versus closed-source models.

## Datasets

- **MathVista** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions in the dataset.

## Input / output format

**Input**: Multimodal instances consisting of an image and a text question/prompt. For TTS methods, the same instance is processed using different inference strategies (Zero-Shot, CoT, Best-of-N, Self-Consistency, Beam Search, Self-Refinement).

**Output**: Text-based answer (typically multiple-choice or short-form) generated by the VLM. For verification-based methods, an external verifier model may also produce a ranking or selection.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if normalize_answer(pred) == normalize_answer(gold))
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Assuming test-time scaling methods like Self-Refinement or CoT universally improve performance; they often degrade open-source models or perception-heavy benchmarks.
- Attempting to use internal confidence scores for closed-source models (e.g., Gemini, GPT-4o), which do not expose token-level probabilities.
- Expecting stochastic decoding to help on deterministic visual tasks where models are already near saturation.

## Evidence (verbatim from paper)

> The evaluation metrics for all datasets are accuracy scores, representing the percentage of correctly answered questions.

## Citation

```bibtex
@misc{ahmadpour2025testtimescaling,
  title={Limits and Gains of Test-Time Scaling in Vision-Language Reasoning},
  author={Ahmadpour et al. (2025)},
  year={2025},
  note={arXiv:2512.11109}
}
```

- arXiv: 2512.11109