Spatial Reasoning Eval

---
name: spatial-reasoning-eval
description: This evaluation probes a model's ability to perform complex spatial reasoning and perspective-taking across single and multiple images. It specifically tests whether the model can correctly establish geometric reference frames, handle multi-step transformations, and generalize across different spatial logic tasks without relying on dataset-specific biases. Use when the user wants to benchmark on MMSI-Bench, MindCube-tiny, OmniSpatial, SPBench, CV-Bench, or asks about evaluating this task. Reports overall accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.22659
  bibtex_key: chen2025geometricallyconstrained
  confidence: high
---

# spatial-reasoning-eval

> Geometrically-Constrained Agent for Spatial Reasoning — Chen et al. (2025) (arXiv:2511.22659, 2025)

## What this evaluates

This evaluation probes a model's ability to perform complex spatial reasoning and perspective-taking across single and multiple images. It specifically tests whether the model can correctly establish geometric reference frames, handle multi-step transformations, and generalize across different spatial logic tasks without relying on dataset-specific biases.

## Datasets

- **MMSI-Bench** — total ?; splits: test (-1)
- **MindCube-tiny** — total ?; splits: test (-1)
- **OmniSpatial** — total ?; splits: test (-1)
- **SPBench** — total ?; splits: test (-1)
- **CV-Bench** — total ?; splits: test (-1)

## Metrics

- `overall accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of instances in the benchmark.

## Input / output format

**Input**: Single or multiple images paired with spatial reasoning questions/prompts (e.g., 4-choice multiple choice questions for MMSI-Bench).

**Output**: A predicted answer choice or generated response corresponding to the spatial reasoning query.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models often default to flawed semantic shortcuts or lossy spatial imagination instead of establishing formal geometric constraints.
- Training-based methods frequently exhibit strong in-domain performance but fail to generalize to out-of-domain benchmarks due to dataset-specific biases.
- Unconstrained tool integration without explicit task constraints leads to unreliable planning and incorrect geometric computations.

## Evidence (verbatim from paper)

> For all benchmarks, we report both overall accuracy (%) and subcategory accuracy (%).

## Citation

```bibtex
@misc{chen2025geometricallyconstrained,
  title={Geometrically-Constrained Agent for Spatial Reasoning},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2511.22659}
}
```

- arXiv: 2511.22659