E3vs Bench Eval

---
name: e3vs-bench-eval
description: Probes 5-DoF viewpoint control and active perception in photorealistic 3D scenes. Tests whether vision-language models can navigate, resolve occlusions, and answer questions by strategically selecting viewpoints to gather spatially dependent visual evidence. Use when the user wants to benchmark on E3VS-Bench, or asks about evaluating this task. Reports VLM Judge Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17969
  bibtex_key: sakamoto2026e3vsbench
  confidence: high
---

# e3vs-bench-eval

> E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes — Sakamoto et al. (2026) (arXiv:2604.17969, 2026)

## What this evaluates

Probes 5-DoF viewpoint control and active perception in photorealistic 3D scenes. Tests whether vision-language models can navigate, resolve occlusions, and answer questions by strategically selecting viewpoints to gather spatially dependent visual evidence.

## Datasets

- **E3VS-Bench** — total 2014; splits: train (1406), val (231), test (377)

## Metrics

- `VLM Judge Score` **(primary)** — range: [1, 5]
  - Score of 1 or 5 assigned by a GPT-5.1 judge comparing the predicted response to the ground-truth answer and end/goal images. Validated with Spearman ρ=0.54 against human evaluation.
- `Average Steps` — range: steps
  - Mean number of navigation steps taken per episode across the test set.
- `Collision Rate` — range: [0, 1]
  - Fraction of episodes where at least one collision occurs. Calculated as binary indicator (1 if collision, 0 otherwise) averaged over all episodes.

## Input / output format

**Input**: Current viewpoint image(s) at 512×512 resolution with 90° FOV, task question, and optionally a history of previous frames (1, 3, or 5) for memory ablation studies.

**Output**: Action command (move_forward 0.25m or rotate 30°) or final answer. Max token length is 128 without reasoning or 256+ with reasoning. Defaults to move_forward if no action command is generated within the token limit.

## Scoring recipe

```python
# VLM Judge Score
judge = GPT51Judge()
score = judge.evaluate(prediction, ground_truth, end_image, goal_image) # returns 1 or 5

# Average Steps
avg_steps = sum(len(trajectory) for trajectory in trajectories) / len(trajectories)

# Collision Rate
collision_rate = sum(1 if episode_has_collision(ep) else 0 for ep in episodes) / len(episodes)
```

## Common pitfalls

- GPT-5.1 was used for dataset filtering, so it is excluded from blind and start-view baselines to prevent evaluation bias.
- Models may exploit dataset bias by answering from the initial viewpoint without exploring, particularly for binary tasks like OST.
- The max 25-step limit and default move_forward fallback can artificially inflate collision rates or truncate valid exploration trajectories.

## Evidence (verbatim from paper)

> Regarding answer correctness, we employ a VLM-as-a-judge framework in accordance with OpenEQA, using GPT 5.1 as the evaluator. The judge VLM receives the predicted response and ground-truth answer, along with the end and goal images, and outputs a score of 5 for correct predictions and 1 for incorrect ones. Furthermore, exploration efficiency is quantified by the average number of steps, while navigation safety is measured using Collision Rate, defined as a binary indicator that takes the value 1 if a collision occurs at least once within an episode and 0 otherwise.

## Citation

```bibtex
@misc{sakamoto2026e3vsbench,
  title={E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes},
  author={Sakamoto et al. (2026)},
  year={2026},
  note={arXiv:2604.17969}
}
```

- arXiv: 2604.17969