Visual Spatial Reasoning Eval

---
name: visual-spatial-reasoning-eval
description: This benchmark evaluates visual language models' ability to understand and reason about spatial relationships between objects in images. It specifically probes orientation-dependent relations, frame-of-reference shifts (intrinsic vs. relative), and zero-shot generalization to unseen object concepts. Use when the user wants to benchmark on VSR, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.00363
  bibtex_key: liu2022visualspatialreasoning
  confidence: high
---

# visual-spatial-reasoning-eval

> Visual Spatial Reasoning — Liu et al. (2022) (arXiv:2205.00363, 2022)

## What this evaluates

This benchmark evaluates visual language models' ability to understand and reason about spatial relationships between objects in images. It specifically probes orientation-dependent relations, frame-of-reference shifts (intrinsic vs. relative), and zero-shot generalization to unseen object concepts.

## Datasets

- **VSR** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/cambridgeltl/visual-spatial-reasoning

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted True/False labels out of the total number of test instances. Calculated as (number of correct predictions) / (total predictions).

## Input / output format

**Input**: Image-text pairs, where the text is a caption describing a spatial relation between objects in the image.

**Output**: Binary classification: True or False, indicating whether the caption accurately describes the spatial relation in the image.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Model performance is sensitive to random seeds; results should be averaged over at least three runs with different seeds.
- The zero-shot split is significantly harder than the random split due to unseen object concepts, not just fewer training examples.
- Proximity relations (e.g., 'near', 'far') are highly concept-dependent and may plateau or drop to chance level in zero-shot settings.

## Evidence (verbatim from paper)

> As shown in Table 4, the best-performing models on the random split are LXMERT and ViLT, reaching around 70% accuracy while VisualBERT is just slightly better than the chance level. On the zero-shot split, all models’ performance decline substantially and the best model ViLT only obtains 63.0% accuracy.

## Citation

```bibtex
@misc{liu2022visualspatialreasoning,
  title={Visual Spatial Reasoning},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2205.00363}
}
```

- arXiv: 2205.00363