Spatial457 Eval

---
name: spatial457-eval
description: This benchmark evaluates large multimodal models' ability to perform 6D spatial reasoning across multiple difficulty levels. It probes capabilities including multi-object recognition, 2D and 3D location understanding, 3D orientation interpretation, and occlusion/collision prediction. It also quantifies systematic prediction biases across visual attributes like color, shape, size, and pose. Use when the user wants to benchmark on Spatial457, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.08636
  bibtex_key: wang2025spatial457
  confidence: high
---

# spatial457-eval

> Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models — Wang et al. (2025) (arXiv:2502.08636, 2025)

## What this evaluates

This benchmark evaluates large multimodal models' ability to perform 6D spatial reasoning across multiple difficulty levels. It probes capabilities including multi-object recognition, 2D and 3D location understanding, 3D orientation interpretation, and occlusion/collision prediction. It also quantifies systematic prediction biases across visual attributes like color, shape, size, and pose.

## Datasets

- **Spatial457** — total 1000; splits: test (1000); repo https://github.com/XingruiWang/Spatial457

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions per category or difficulty level.
- `RPDR` — range: percent
  - Relative Performance Dropping Rate; quantifies the decline in accuracy when introducing additional spatial factors or complexity levels.
- `CV` — range: other
  - Coefficient of Variation for prediction bias; calculated as the standard deviation divided by the mean of the predicted probability distribution for each attribute label.

## Input / output format

**Input**: A synthetic 3D-rendered image paired with a natural language question probing spatial relationships, object attributes, or pose.

**Output**: A text response containing the predicted answer (and optionally reasoning steps) to the posed question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for p, g in zip(predictions, gold_answers):
        if normalize_text(p) == normalize_text(g):
            correct += 1
    return (correct / len(gold_answers)) * 100

def compute_cv(attribute_predictions):
    # attribute_predictions: dict mapping label -> count/prob
    probs = list(attribute_predictions.values())
    mean_p = sum(probs) / len(probs)
    std_p = (sum((x - mean_p)**2 for x in probs) / len(probs)) ** 0.5
    return std_p / mean_p if mean_p > 0 else 0
```

## Common pitfalls

- Models exhibit strong attribute biases (e.g., favoring 'yellow' for color or 'front' for pose), which can artificially inflate accuracy if the dataset were not synthetically balanced.
- Performance drops sharply from 2D to 3D/6D tasks; evaluating only on 2D spatial relationships will significantly overestimate a model's true spatial reasoning capability.
- The benchmark uses synthetic 3D renders; results may not directly transfer to real-world photographic inputs without domain adaptation.

## Evidence (verbatim from paper)

> We generate 1,000 images to test the performance of large vision-language models. For the seven types of questions, we generate 670 single-object questions (L1-Single), 4,995 multiple-object questions (L2-Multi-obj.), 4,995 2D spatial relationship questions (L3-2D-Spatial), 3,534 occlusion questions (L4-Occlusion), 4,555 3D pose questions (L4-3D-Pose), 3,166 collision questions (L5-Collision), and 1,837 6D spatial relationship questions (L5-6D-Spatial)... The RPDR analysis provides a structured approach to quantify the decline in accuracy with the introduction of each new factor, as detailed in [Tab. 3].

## Citation

```bibtex
@misc{wang2025spatial457,
  title={Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.08636}
}
```

- arXiv: 2502.08636