Spatial Reasoning Vqa Eval

---
name: spatial-reasoning-vqa-eval
description: Evaluates a vision-language model's ability to perform spatial reasoning tasks, including relative positioning, counting, size comparison, and cross-dataset generalization. It probes whether models learn transferable spatial concepts rather than memorizing dataset-specific patterns or visual artifacts. Use when the user wants to benchmark on GRAID-BDD, GRAID-NuImages, BLINK, A-OKVQA, NaturalBench, RealWorldQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.22118
  bibtex_key: [REDACTED]
  confidence: high
---

# spatial-reasoning-vqa-eval

> GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation — Elmaaroufi et al. (2025) (arXiv:2510.22118, 2025)

## What this evaluates

Evaluates a vision-language model's ability to perform spatial reasoning tasks, including relative positioning, counting, size comparison, and cross-dataset generalization. It probes whether models learn transferable spatial concepts rather than memorizing dataset-specific patterns or visual artifacts.

## Datasets

- **GRAID-BDD** — total ?; splits: train (-1), test (1000)
- **GRAID-NuImages** — total ?; splits: test (1000)
- **BLINK** — total ?; splits: test (-1)
- **A-OKVQA** — total ?; splits: test (-1)
- **NaturalBench** — total ?; splits: test (-1)
- **RealWorldQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard exact-match accuracy: the model's predicted answer is compared to the ground-truth answer, with 1 point awarded for a match and 0 otherwise. Aggregated as a percentage over the test set.

## Input / output format

**Input**: RGB image paired with a natural language question requiring spatial reasoning (e.g., relative position, counting, size comparison).

**Output**: Textual answer or selected option from a multiple-choice set, as generated by the VLM.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models may overfit to specific question templates or dataset-specific visual distributions (e.g., training on car-heavy GRAID-BDD but evaluating on diverse BLINK scenes).
- Using different evaluation frameworks or decoding strategies can cause inconsistent results; the authors explicitly switched to VLMEvalKit to match reported baselines.
- Adversarial benchmarks like NaturalBench are designed to expose memorization; stable scores here indicate genuine spatial reasoning rather than shortcut learning.

## Evidence (verbatim from paper)

> We observe a significant 32.5% improvement on A-OKVQA and 15.94% overall improvement on BLINK, with particularly impressive gains on core spatial reasoning tasks: +41.13% on Relative Depth, +31.98% on Visual Correspondence, and +30.77% on Spatial Relations.

## Citation

```bibtex
@misc{elmaaroufi2025graid,
  title={GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation},
  author={Elmaaroufi et al. (2025)},
  year={2025},
  note={arXiv:2510.22118}
}
```

- arXiv: 2510.22118