Gsr Bench Eval

---
name: gsr-bench-eval
description: Evaluates multimodal LLMs' ability to understand and disambiguate spatial relations (e.g., on, under, left of, right of, in front of, behind) between objects in images. It isolates spatial reasoning from object grounding by providing depth maps, bounding boxes, and segmentation masks alongside images. Use when the user wants to benchmark on GSR-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.13246
  bibtex_key: rajabi2024gsrbench
  confidence: high
---

# gsr-bench-eval

> GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs — Rajabi et al. (2024) (arXiv:2406.13246, 2024)

## What this evaluates

Evaluates multimodal LLMs' ability to understand and disambiguate spatial relations (e.g., on, under, left of, right of, in front of, behind) between objects in images. It isolates spatial reasoning from object grounding by providing depth maps, bounding boxes, and segmentation masks alongside images.

## Datasets

- **GSR-Bench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted spatial relations out of total test instances.

## Input / output format

**Input**: RGB image, spatial relation query/clause (e.g., 'Is object A on object B?'), and optionally depth map, bounding box coordinates, and segmentation masks.

**Output**: Text prediction of the spatial relation (e.g., 'on', 'under', 'left of', 'right of', 'in front of', 'behind').

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Confusing spatial relations when depth cues are absent, leading to front/back or left/right disambiguation errors.
- Models may rely on object grounding priors rather than true spatial reasoning if masks/boxes are not properly conditioned.
- Prompting strategies and depth-augmentation significantly impact scores, making cross-model comparisons sensitive to input formatting.

## Evidence (verbatim from paper)

> LLaMA-3-LLaVA-NeXT-8B achieves 86.1% accuracy, outperforming XVLM-COCO (60.4%) and demonstrating strong scalability with depth-augmented prompting that improves disambiguation of front/back and left/right relations.

## Citation

```bibtex
@misc{rajabi2024gsrbench,
  title={GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs},
  author={Rajabi et al. (2024)},
  year={2024},
  note={arXiv:2406.13246}
}
```

- arXiv: 2406.13246