Spatialthinker Eval

---
name: spatialthinker-eval
description: Evaluates multimodal LLMs on 3D spatial reasoning, depth/distance estimation, and general visual question answering. It probes the model's ability to ground objects in 3D space, understand spatial relations, and generalize to real-world VQA tasks using only RGB inputs. Use when the user wants to benchmark on SpatialThinker Evaluation Suite (12 VQA Benchmarks), or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.07403
  bibtex_key: batra2025spatialthinker
  confidence: high
---

# spatialthinker-eval

> SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards — Hunar Batra et al. (arXiv:2511.07403, 2025)

## What this evaluates

Evaluates multimodal LLMs on 3D spatial reasoning, depth/distance estimation, and general visual question answering. It probes the model's ability to ground objects in 3D space, understand spatial relations, and generalize to real-world VQA tasks using only RGB inputs.

## Datasets

- **SpatialThinker Evaluation Suite (12 VQA Benchmarks)** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Computed as the number of exact matches between the model's generated answer and the ground-truth answer divided by the total number of instances.

## Input / output format

**Input**: RGB image paired with a natural language question.

**Output**: Natural language text answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Evaluated strictly zero-shot with greedy decoding (temperature=0.0), not few-shot or with sampling.
- SpatialRGPT uses depth inputs while all other baselines use only RGB, creating an input modality mismatch.
- Proprietary models (GPT-4o, Claude) use their default system prompts, which may differ from open-source model prompting.

## Evidence (verbatim from paper)

> All models are evaluated in a zero-shot setting using greedy decoding (temperature = 0.0). Accuracy is the primary evaluation metric.

## Citation

```bibtex
@misc{batra2025spatialthinker,
  title={SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards},
  author={Hunar Batra et al.},
  year={2025},
  note={arXiv:2511.07403}
}
```

- arXiv: 2511.07403