Vsi Bench Eval

---
name: vsi-bench-eval
description: Evaluates multimodal large language models' ability to reason about 3D spatial relationships from visual inputs. It probes capabilities across configurational tasks (counting, relative distance/direction, route planning), measurement estimation (object/room size, absolute distance), and spatiotemporal ordering. Use when the user wants to benchmark on VSI-Bench, or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.08592
  bibtex_key: [REDACTED]
  confidence: high
---

# vsi-bench-eval

> Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations — Yuan et al. (2026) (arXiv:2603.08592, 2026)

## What this evaluates

Evaluates multimodal large language models' ability to reason about 3D spatial relationships from visual inputs. It probes capabilities across configurational tasks (counting, relative distance/direction, route planning), measurement estimation (object/room size, absolute distance), and spatiotemporal ordering.

## Datasets

- **VSI-Bench** — total ?; splits: test (-1)

## Metrics

- `score` **(primary)** — range: percent
  - Percentage of correctly answered questions per subtask (object counting, distance, direction, size, route planning, order of appearance), averaged across all subtasks for the overall score.

## Input / output format

**Input**: Annotated images with numeric object IDs, textual references encoding 3D geometric attributes (bounding boxes, shape primitives, room size), and a prompt template specifying coordinate system handedness and upright orientation.

**Output**: Natural language answers to spatial reasoning questions, often requiring step-by-step reasoning or direct numerical/categorical responses.

## Scoring recipe

```python
def compute_vsi_bench_score(predictions, gold):
    task_scores = {}
    for task in ['obj_count', 'abs_dist', 'obj_size', 'room_size', 'rel_dist', 'rel_dir', 'route_plan', 'appr_order']:
        correct = sum(1 for p, g in zip(predictions[task], gold[task]) if p == g)
        task_scores[task] = (correct / len(gold[task])) * 100
    avg_score = sum(task_scores.values()) / len(task_scores)
    return task_scores, avg_score
```

## Common pitfalls

- Voxel-based object separation struggles in cluttered scenes, leading to inaccurate object counts and size estimations.
- Point clouds reconstructed from images are noisy, often causing overestimated bounding box dimensions.
- Benchmarks assume dense video frames, but real-world scenarios often provide only sparse views, requiring robust multi-view reasoning.

## Evidence (verbatim from paper)

> Among all MLLMs, GPT-5 achieves the highest overall score. While the fine-tuned VG LLM attains a comparable performance, it should be noted that its training data include videos overlapping with VSI-Bench and questions similar to those in the benchmark.

## Citation

```bibtex
@misc{yuan2026boosting,
  title={Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations},
  author={Yuan et al. (2026)},
  year={2026},
  note={arXiv:2603.08592}
}
```

- arXiv: 2603.08592