Spatial Intelligence Eval

---
name: spatial-intelligence-eval
description: Evaluates a model's ability to perceive, reason about, and reconstruct 3D spatial layouts, multi-view relationships, and perspective-taking from visual inputs. It probes robustness against language shortcuts and tests generalization to longer video sequences and embodied manipulation tasks. Use when the user wants to benchmark on VSI-Bench, MMSI-Bench, MindCube, ViewSpatial-Bench, SITE, MMBench-En, EmbodiedBench (spatial subset), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.13719
  bibtex_key: cai2025scalingspatial
  confidence: high
---

# spatial-intelligence-eval

> Scaling Spatial Intelligence with Multimodal Foundation Models — Cai et al. (2025) (arXiv:2511.13719, 2025)

## What this evaluates

Evaluates a model's ability to perceive, reason about, and reconstruct 3D spatial layouts, multi-view relationships, and perspective-taking from visual inputs. It probes robustness against language shortcuts and tests generalization to longer video sequences and embodied manipulation tasks.

## Datasets

- **VSI-Bench** — total ?; splits: test (-1)
- **MMSI-Bench** — total ?; splits: test (-1)
- **MindCube** — total ?; splits: train (10000), test (-1)
- **ViewSpatial-Bench** — total ?; splits: test (-1)
- **SITE** — total ?; splits: test (-1)
- **MMBench-En** — total ?; splits: test (-1)
- **EmbodiedBench (spatial subset)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard multiple-choice accuracy: the percentage of questions where the model's predicted option exactly matches the ground-truth label.
- `success rate` — range: percent
  - The proportion of robot manipulation tasks completed successfully within the allowed steps or constraints.

## Input / output format

**Input**: Visual inputs (images or uniformly sampled video frames, e.g., 32 frames for VSI-Bench) paired with multiple-choice questions or spatial instructions. For robot tasks, images with bounding-box coordinates or enriched spatial prompts.

**Output**: Multiple-choice letter (A/B/C/D) or binary success/failure for robot manipulation tasks.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().upper() == g.strip().upper())
    return correct / len(gold) * 100

def compute_success_rate(success_flags):
    return sum(success_flags) / len(success_flags) * 100
```

## Common pitfalls

- Models may exploit language shortcuts or text priors instead of visual reasoning (mitigated by VSI-Debiased and no-vision tests).
- Performance can be artificially inflated by overfitting to answer option ordering (addressed via soft/hard circular tests).
- Training on short video sequences (≤16 frames) may not guarantee robust extrapolation to longer contexts at inference.

## Evidence (verbatim from paper)

> We report success rates under two prompting settings: the official prompt (OP) and a spatial-intelligence-oriented prompt (SIP). OP supplies bounding-box coordinates extracted from the input image, whereas SIP enriches OP with additional object-grounding cues to reduce ambiguity from object recognition and better isolate spatial-reasoning performance. Across both OP and SIP, SenseNova-SI delivers substantial improvements, demonstrating that enhanced spatial intelligence directly benefits embodied manipulation.

## Citation

```bibtex
@misc{cai2025scalingspatial,
  title={Scaling Spatial Intelligence with Multimodal Foundation Models},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2511.13719}
}
```

- arXiv: 2511.13719