Omnispatial Eval

---
name: omnispatial-eval
description: This benchmark evaluates vision-language models on advanced spatial reasoning capabilities, specifically probing dynamic reasoning, complex spatial logic, spatial interaction, and perspective-taking. It measures how well models understand and manipulate spatial relationships, temporal changes, and viewpoint shifts beyond basic object recognition. Use when the user wants to benchmark on OmniSpatial, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.03135
  bibtex_key: jia2025omnispatial
  confidence: high
---

# omnispatial-eval

> OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models — Jia et al. (2025) (arXiv:2506.03135, 2025)

## What this evaluates

This benchmark evaluates vision-language models on advanced spatial reasoning capabilities, specifically probing dynamic reasoning, complex spatial logic, spatial interaction, and perspective-taking. It measures how well models understand and manipulate spatial relationships, temporal changes, and viewpoint shifts beyond basic object recognition.

## Datasets

- **OmniSpatial** — total 8400; splits: train (6900), test (-1); repo https://github.com/qizekun/OmniSpatial

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. For standard models, correctness is determined via direct answer matching, regular-expression parsing, JSON parsing, or LLM-as-a-Judge. For reasoning models with unstructured chain-of-thought outputs, an external judge (GPT-4.1-mini) compares the final answer against ground truth.

## Input / output format

**Input**: An image paired with a multiple-choice question requiring spatial reasoning.

**Output**: A selected option (e.g., A, B, C, D) and optionally a chain-of-thought reasoning trace.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred == gold:
            correct += 1
        elif is_reasoning_model:
            if llm_judge(pred, gold):
                correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Unstructured chain-of-thought outputs from reasoning models cannot be reliably parsed with regex; an LLM-as-a-Judge is required to extract the final answer.
- Evaluation scores vary significantly depending on the output protocol used (direct answer vs. regex vs. JSON vs. LLM judge), so consistency in parsing strategy is critical.
- Models often default to ego-centric viewpoints, leading to artificially low scores on perspective-taking tasks unless explicitly prompted for novel-view reasoning.

## Evidence (verbatim from paper)

> We measure accuracy on multiple-choice questions. For standard proprietary and open-source models, we test four output protocols: direct answer, regular-expression parsing, JSON parsing, and LLM-as-a-Judge*(Zheng et al., [2023])*. For reasoning-oriented models with unstructured CoT outputs, correctness is assessed by GPT-4.1-mini against ground truth.

## Citation

```bibtex
@misc{jia2025omnispatial,
  title={OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models},
  author={Jia et al. (2025)},
  year={2025},
  note={arXiv:2506.03135}
}
```

- arXiv: 2506.03135