Vbvr Bench Eval

---
name: vbvr-bench-eval
description: Systematic video reasoning capabilities grounded in five cognitive faculties: perception, transformation, spatiality, abstraction, and knowledge. It probes spatiotemporal reasoning, mental manipulation, and rule-based problem solving on video sequences. Use when the user wants to benchmark on VBVR-Dataset, or asks about evaluating this task. Reports rule-based scorer.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.20159
  bibtex_key: wang2026verybigvideoreasoning
  confidence: high
---

# vbvr-bench-eval

> A Very Big Video Reasoning Suite — Maijunxian Wang et al. (2026) (arXiv:2602.20159, 2026)

## What this evaluates

Systematic video reasoning capabilities grounded in five cognitive faculties: perception, transformation, spatiality, abstraction, and knowledge. It probes spatiotemporal reasoning, mental manipulation, and rule-based problem solving on video sequences.

## Datasets

- **VBVR-Dataset** — total 1007500; splits: train (1000000), test (7500)

## Metrics

- `rule-based scorer` **(primary)** — range: [0, 1]
  - Deterministic verification score comparing model predictions against ground-truth solution trajectories. Additionally, Spearman rank correlation (ρ) is computed to measure alignment between automated scores and human judgments.

## Input / output format

**Input**: A single initial frame image (first_frame.png) and a text prompt (prompt.txt) describing the task instruction.

**Output**: Model predictions of the reasoning trajectory or final state, evaluated against ground_truth.mp4 (complete solution trajectory) and final_frame.png (target state).

## Scoring recipe

```python
def score(predictions, gold):
    # Verify prediction against ground truth trajectory
    rule_score = verify_trajectory(predictions, gold['ground_truth.mp4'])
    # Compute alignment with human judgments
    human_scores = get_human_ratings(predictions)
    spearman_rho = scipy.stats.spearmanr(rule_score, human_scores).correlation
    return rule_score, spearman_rho
```

## Common pitfalls

- Data leakage between train and test splits if random seeds are not strictly disjoint.
- Attempting to solve tasks using only the first frame, violating the explicit 'video dependency' constraint designed to test temporal reasoning.
- Ignoring the deterministic solvability requirement, which mandates unique verifiable success criteria per task.

## Evidence (verbatim from paper)

> VBVR generates one million training samples across 100 training tasks (10,000 per task) and 7500 test samples across 150 test tasks (50 per task). Training and test splits are constructed using disjoint random seed ranges to prevent data leakage. ... Each generator deterministically produces a four-component output: (1) first_frame.png (initial state), (2) prompt.txt (task instruction), (3) final_frame.png (target state), and (4) ground_truth.mp4 (complete solution trajectory). VBVR-Bench, a verifiable evaluation framework using rule-based scorers aligned with human judgments (Spearman ρ > 0.9), revealing significant performance gaps between current models and human-level reasoning.

## Citation

```bibtex
@misc{wang2026verybigvideoreasoning,
  title={A Very Big Video Reasoning Suite},
  author={Maijunxian Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.20159}
}
```

- arXiv: 2602.20159