Spce 10 Eval

---
name: spce-10-eval
description: This benchmark evaluates multimodal large language models on compositional spatial intelligence by testing their ability to reason across 10 atomic spatial capabilities (e.g., counting, localization, spatial relations) combined into 8 complex tasks. It probes scene understanding using 2D and 3D inputs through multiple-choice questions, revealing how models handle hierarchical spatial reasoning and capability integration. Use when the user wants to benchmark on SpaCE-10, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.07966
  bibtex_key: gong2025spce10
  confidence: high
---

# spce-10-eval

> SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence — Gong et al. (2025) (arXiv:2506.07966, 2025)

## What this evaluates

This benchmark evaluates multimodal large language models on compositional spatial intelligence by testing their ability to reason across 10 atomic spatial capabilities (e.g., counting, localization, spatial relations) combined into 8 complex tasks. It probes scene understanding using 2D and 3D inputs through multiple-choice questions, revealing how models handle hierarchical spatial reasoning and capability integration.

## Datasets

- **SpaCE-10** — total 5000; splits: test (5000); repo https://github.com/VisionXLab/SpaCE-10

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total questions) * 100. Applied uniformly to both single-answer and multiple-choice formats.

## Input / output format

**Input**: Visual inputs (2D images or 3D point clouds) paired with multiple-choice questions targeting specific spatial reasoning capabilities.

**Output**: A selected answer option from the provided choices (single-answer or multiple-answer format).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Random baseline is 25% for 4-choice questions, but the paper notes performance might be lower than random due to removing random assignment in LMMs-Eval.
- Smaller models tend to overfit to single-answer formats, drastically underperforming on multiple-answer tasks (<5% vs >30%).
- High overall accuracy does not necessarily reflect strong atomic spatial capabilities; capability breakdown is needed to avoid misleading conclusions.

## Evidence (verbatim from paper)

> For the 3 questions with integrating C7 (Multi-view Fusion) capability, the models’ accuracy in SA, OO, and OS tasks decreases by 19.1%, 7.6%, and 21.0%, respectively.

## Citation

```bibtex
@misc{gong2025spce10,
  title={SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence},
  author={Gong et al. (2025)},
  year={2025},
  note={arXiv:2506.07966}
}
```

- arXiv: 2506.07966