Spatialqa Eval

---
name: spatialqa-eval
description: Evaluates the ability of vision-language models to perform multi-step spatial logical reasoning by tracking object dependencies and understanding scene layouts across real-world indoor environments. Use when the user wants to benchmark on SpatiaLQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.20901
  bibtex_key: xie2026spatialqa
  confidence: medium
---

# spatialqa-eval

> SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models — Xie et al. (2026) (arXiv:2602.20901, 2026)

## What this evaluates

Evaluates the ability of vision-language models to perform multi-step spatial logical reasoning by tracking object dependencies and understanding scene layouts across real-world indoor environments.

## Datasets

- **SpatiaLQA** — total 9605; splits: test (-1); repo https://github.com/xieyc99/SpatiaLQA

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy for closed-form QA: 1 if the model's predicted answer matches the ground truth answer (case-insensitive, normalized whitespace), 0 otherwise. Averaged over all 9,605 samples.

## Input / output format

**Input**: A single RGB image of an indoor scene and a natural language question describing a spatial relationship or logical query about objects within that scene.

**Output**: A natural language text answer corresponding to the question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize(pred) == normalize(gold):
            correct += 1
    return correct / len(gold_answers)
```

## Common pitfalls

- Counting different camera angles or lighting changes of the same physical setup as distinct scenes, which violates the dataset's scene definition rules.
- Allowing the same object to appear more than ten times within a single scene, which breaks the designed diversity and frequency constraints.
- Failing to verify step validity and prerequisite correctness during the annotation review cycle, leading to logically unsound QA pairs.

## Evidence (verbatim from paper)

> The benchmark comprises 9,605 QA pairs from 241 real-world indoor scenes, constructed via manual annotation and graph-augmentation techniques. Each sample annotation consists of four components: the question, the answer, the corresponding image path, and the associated scene category.

## Citation

```bibtex
@misc{xie2026spatialqa,
  title={SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models},
  author={Xie et al. (2026)},
  year={2026},
  note={arXiv:2602.20901}
}
```

- arXiv: 2602.20901