Spatial Dise Eval

---
name: spatial-dise-eval
description: Evaluates spatial reasoning capabilities in vision-language models across a 2x2 cognitive taxonomy (Intrinsic/Extrinsic × Static/Dynamic). It probes mental rotation, multi-step 3D transformations, and dynamic scene simulation using synthetically rendered 3D VQA pairs. Use when the user wants to benchmark on Spatial-DISE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13394
  bibtex_key: huang2025spatialdise
  confidence: high
---

# spatial-dise-eval

> Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models — Huang et al. (2025) (arXiv:2510.13394, 2025)

## What this evaluates

Evaluates spatial reasoning capabilities in vision-language models across a 2x2 cognitive taxonomy (Intrinsic/Extrinsic × Static/Dynamic). It probes mental rotation, multi-step 3D transformations, and dynamic scene simulation using synthetically rendered 3D VQA pairs.

## Datasets

- **Spatial-DISE** — total 12000; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Multiple-choice accuracy computed via exact match between the model's predicted option and the ground truth answer. Malformed or verbose model outputs are first parsed using an external LLM (Deepseek-R1) before comparison.

## Input / output format

**Input**: 3D scene image (synthetically rendered via Blender) paired with a multiple-choice question text probing spatial relationships, transformations, or mental simulation.

**Output**: A single letter/option corresponding to the correct answer, optionally preceded by reasoning text.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        parsed_pred = parse_answer(pred)  # e.g., via Deepseek-R1
        if parsed_pred == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Model outputs are frequently malformed or verbose, requiring an external parser (Deepseek-R1) before exact-match scoring can be reliably applied.
- Human baseline performance is averaged across a minimum of three unique participants per question, introducing inter-subject variance not captured by standard automated metrics.
- Fine-tuning on the Spatial-DISE dataset can cause catastrophic forgetting on other spatial benchmarks (e.g., CVBench, SAT), complicating claims about generalization.

## Evidence (verbatim from paper)

> We evaluate multiple-choice accuracy using exact match via the VLMEvalKit. Deepseek-R1 is used to parse answers from malformed model outputs.

## Citation

```bibtex
@misc{huang2025spatialdise,
  title={Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2510.13394}
}
```

- arXiv: 2510.13394