Perceptionprocessbench Eval

---
name: perceptionprocessbench-eval
description: Evaluates a vision-language process reward model's ability to detect step-level visual grounding and reasoning errors in structured multimodal reasoning traces. The benchmark specifically probes whether the model can distinguish between correct and subtly mutated perception steps that are designed to be challenging for automated error detection. Use when the user wants to benchmark on PerceptionProcessBench, or asks about evaluating this task. Reports step-level correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.23250
  bibtex_key: ong2025vlprm
  confidence: high
---

# perceptionprocessbench-eval

> Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned — Ong et al. (2025) (arXiv:2509.23250, 2025)

## What this evaluates

Evaluates a vision-language process reward model's ability to detect step-level visual grounding and reasoning errors in structured multimodal reasoning traces. The benchmark specifically probes whether the model can distinguish between correct and subtly mutated perception steps that are designed to be challenging for automated error detection.

## Datasets

- **PerceptionProcessBench** — total 2000; splits: test (2000); repo https://github.com/theogbrand/vlprm

## Metrics

- `step-level correctness` **(primary)** — range: [0, 1]
  - Proportion of correctly classified steps (correct vs. incorrect) out of the total number of evaluated steps. Computed as (number of correct predictions) / (total steps).

## Input / output format

**Input**: Image $I$, question $q$, preceding reasoning steps $s_{<i}$, and candidate step $s_i$.

**Output**: Binary label: 'correct' or 'incorrect'.

## Scoring recipe

```python
correct = 0
total = 0
for step in PerceptionProcessBench:
    pred = model.predict(step.image, step.question, step.context, step.step)
    if pred == step.gold_label:
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Synthetic negative examples are deliberately mutated to be subtle, making them significantly harder to detect than obvious errors.
- The underlying training data is highly imbalanced (86% perception errors), so models may bias towards predicting 'correct' if not properly calibrated or balanced during evaluation.

## Evidence (verbatim from paper)

> assessing their performance on perception error detection requires a dataset with explicit correctness annotations for perception steps. To address this, we synthetically construct PerceptionProcessBench from PuzzleVQA.

## Citation

```bibtex
@misc{ong2025vlprm,
  title={Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned},
  author={Ong et al. (2025)},
  year={2025},
  note={arXiv:2509.23250}
}
```

- arXiv: 2509.23250