Revqa Spatial Reasoning Eval

---
name: revqa-spatial-reasoning-eval
description: Evaluates the spatial reasoning and logical comprehension capabilities of multimodal large language models (MLLMs) on synthetic, spatially precise images. It probes robustness to negations, logical operators (AND/OR), adversarial object substitutions, and complex spatial relationships. Use when the user wants to benchmark on RevQA, or asks about evaluating this task. Reports performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.02231
  bibtex_key: chatterjee2024revision
  confidence: high
---

# revqa-spatial-reasoning-eval

> REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models — Chatterjee et al. (2024) (arXiv:2408.02231, 2024)

## What this evaluates

Evaluates the spatial reasoning and logical comprehension capabilities of multimodal large language models (MLLMs) on synthetic, spatially precise images. It probes robustness to negations, logical operators (AND/OR), adversarial object substitutions, and complex spatial relationships.

## Datasets

- **RevQA** — total 50000; splits: test (50000)

## Metrics

- `performance` **(primary)** — range: percent
  - Percentage of correct binary (Yes/No) predictions across all question types. Calculated as (number of correct answers / total number of questions) * 100.

## Input / output format

**Input**: A synthetic 3D-rendered image paired with a natural language question (yes/no format) testing spatial relationships and logical operators.

**Output**: Binary response: 'Yes' or 'No'.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() in ['yes', 'no'] and pred.strip().lower() == gold.strip().lower():
        correct += 1
    total += 1
accuracy = (correct / total) * 100 if total > 0 else 0
```

## Common pitfalls

- Models may exhibit order bias when answering combined questions; the benchmark mitigates this by randomizing clause order, but evaluators should ensure consistent ordering or account for it.
- Evaluation requires deterministic decoding (temperature=0) to avoid stochasticity masking true reasoning capabilities, especially for negation and adversarial questions.
- Binary responses must be strictly parsed as 'Yes' or 'No'; free-form text generation will break the performance calculation.

## Evidence (verbatim from paper)

> We instruct all models to generate binary responses and set the temperature =0, to remove stochasticity in the generated responses. We present our evaluation results in Table 7 and find that all models have a large gap in performance in reasoning over spatial relationships.

## Citation

```bibtex
@misc{chatterjee2024revision,
  title={REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models},
  author={Chatterjee et al. (2024)},
  year={2024},
  note={arXiv:2408.02231}
}
```

- arXiv: 2408.02231