Vr Bench Eval

---
name: vr-bench-eval
description: This benchmark evaluates the spatial reasoning and trajectory planning capabilities of video generation models and vision-language models through maze-solving tasks. It probes whether models can generate coherent, rule-compliant movement sequences or videos that faithfully navigate complex, multi-type mazes such as regular, irregular, 3D, Sokoban, and trap fields. Use when the user wants to benchmark on VR-Bench, or asks about evaluating this task. Reports MF.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.15065
  bibtex_key: yang2025reasoningvideo
  confidence: medium
---

# vr-bench-eval

> Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks — Cheng Yang et al. (2025) (arXiv:2511.15065, 2025)

## What this evaluates

This benchmark evaluates the spatial reasoning and trajectory planning capabilities of video generation models and vision-language models through maze-solving tasks. It probes whether models can generate coherent, rule-compliant movement sequences or videos that faithfully navigate complex, multi-type mazes such as regular, irregular, 3D, Sokoban, and trap fields.

## Datasets

- **VR-Bench** — total ?; splits: train (-1), val (-1)

## Metrics

- `MF` **(primary)** — range: percent
  - Maze Fidelity: the proportion of generated trajectories or video sequences that successfully navigate from the start to the goal while adhering to maze constraints, verified against an optimal reference path in a simulator.
- `VLM-Score` — range: [0, 5]
  - Rule-compliance score evaluated by an external VLM judge, assessing whether predicted actions or generated videos correctly follow the specific movement rules and topology of each maze type.

## Input / output format

**Input**: For video models: a single initial maze image (preprocessed with model-specific padding and center cropping). For VLMs: an initial observation image $I_{0}$ representing the maze start state.

**Output**: Video models: a generated video clip (typically 8–10 seconds) depicting the agent's movement through the maze. VLMs: a discrete action sequence $a_{\text{pred}} = [a_{1}, \dots, a_{T}]$ (e.g., {up, down, left, right}, graph nodes, or 3D directional moves).

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # Execute predicted actions in simulator to get trajectory
    trajectory = execute_actions_in_simulator(predictions.actions)
    # Check validity against optimal reference
    mf = 100.0 if trajectory_matches_optimal(trajectory, gold.optimal_path) else 0.0
    # VLM judge evaluates rule compliance
    vlm_score = vlm_judge_rule_compliance(predictions.actions, gold.maze_type)
    return {'MF': mf, 'VLM-Score': vlm_score}
```

## Common pitfalls

- Input preprocessing (black-border padding and center cropping) varies across proprietary models and can distort spatial relationships if not handled uniformly.
- VLM-Score relies on an external VLM judge rather than hard simulator validation, introducing potential subjectivity or inconsistency in rule-compliance assessment.
- Action spaces differ significantly across maze types (4-directional, graph-node transitions, 6-directional 3D moves), requiring careful mapping to avoid evaluation mismatches.

## Evidence (verbatim from paper)

> MF and VLM-Score denote Maze Fidelity and the rule-compliance score evaluated by a VLM. Given an initial observation image $I_{0}$, the VLM predicts an action sequence $a_{\text{pred}}\=[a_{1},\dots,a_{T}]$, representing its intended movements in the environment. The actions are sequentially executed in the simulator to verify trajectory validity against the optimal reference $a_{\text{opt}}$.

## Citation

```bibtex
@misc{yang2025reasoningvideo,
  title={Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks},
  author={Cheng Yang et al. (2025)},
  year={2025},
  note={arXiv:2511.15065}
}
```

- arXiv: 2511.15065