Spatial Reasoning Eval

---
name: spatial_reasoning_eval
description: Evaluates vision-language models' ability to count objects and reason about spatial relationships (depth, distance, relative position) in images. It probes segmentation capabilities, attention alignment, and robustness to linguistic variations (out-of-distribution shifts). Use when the user wants to benchmark on CLEVR_CoGenT_ValB, CVBench, Pixmo-Count, Static Spatial Reasoning (SAT), VSR, VC Bench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.13362
  bibtex_key: ji2025enhancing
  confidence: high
---

# spatial_reasoning_eval

> Enhancing Spatial Reasoning in Vision-Language Models via Chain-of-Thought Prompting and Reinforcement Learning — Binbin Ji et al. (arXiv:2507.13362, 2025)

## What this evaluates

Evaluates vision-language models' ability to count objects and reason about spatial relationships (depth, distance, relative position) in images. It probes segmentation capabilities, attention alignment, and robustness to linguistic variations (out-of-distribution shifts).

## Datasets

- **CLEVR_CoGenT_ValB** — total ?; splits: val (-1)
- **CVBench** — total ?; splits: test (-1)
- **Pixmo-Count** — total ?; splits: test (-1)
- **Static Spatial Reasoning (SAT)** — total ?; splits: val (-1)
- **VSR** — total ?; splits: val (-1)
- **VC Bench** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - The percentage of predictions that exactly match the ground truth count or answer.
- `Close-Call Percentage` — range: percent
  - The percentage of incorrect predictions that are off by exactly one: |{i:|ŷ_i−y_i|=1 and ŷ_i≠y_i}| / |{i:ŷ_i≠y_i}| × 100%.
- `Pass@1 accuracy` — range: percent
  - Accuracy computed over a single generation per query.
- `Pass@4 accuracy` — range: percent
  - Accuracy computed as the average correctness over four independent generations per query.

## Input / output format

**Input**: RGB image and a natural language question (e.g., 'Which object is closer to the camera?' or 'How many red cubes are there?').

**Output**: A single predicted answer (e.g., object ID/name or count), or a structured Scene Graph followed by the answer when using Chain-of-Thought prompting.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truths):
    correct = sum(1 for p, g in zip(predictions, ground_truths) if p == g)
    off_by_one = sum(1 for p, g in zip(predictions, ground_truths) if p != g and abs(p - g) == 1)
    incorrect = sum(1 for p, g in zip(predictions, ground_truths) if p != g)
    accuracy = (correct / len(predictions)) * 100
    close_call_pct = (off_by_one / incorrect) * 100 if incorrect > 0 else 0.0
    return accuracy, close_call_pct
```

## Common pitfalls

- Confusing Pass@1 (single generation accuracy) with Pass@4 (average accuracy over 4 samples), which show different trade-offs between SFT and GRPO.
- Overlooking the out-of-distribution (OOD) evaluation protocol, which requires rephrasing test queries from 'close to' to 'far from' to test semantic generalization rather than surface memorization.
- Assuming standard Chain-of-Thought improves performance; the paper finds naive CoT degrades spatial reasoning and requires a structured two-step Scene Graph prompt to avoid reward hacking.

## Evidence (verbatim from paper)

> To test this, we evaluate performance using two complementary metrics on our counting dataset: Accuracy: The percentage of predictions that exactly match the ground truth count. Close-Call Percentage: Defined as |{i:|ŷ_i−y_i|=1 and ŷ_i≠y_i}| / |{i:ŷ_i≠y_i}| × 100%, where ŷ_i is the model’s predicted count and y_i is the ground truth.

## Citation

```bibtex
@misc{ji2025enhancing,
  title={Enhancing Spatial Reasoning in Vision-Language Models via Chain-of-Thought Prompting and Reinforcement Learning},
  author={Binbin Ji et al.},
  year={2025},
  note={arXiv:2507.13362}
}
```

- arXiv: 2507.13362