M Cube Eval

---
name: m-cube-eval
description: Evaluates 3D spatial reasoning and combinatorial planning by requiring models to assemble jigsaw-style pieces into a 5x5x5 cube under physical constraints. The benchmark probes the model's ability to extract geometric patterns from rendered views and logically arrange pieces without gaps or overlaps. Use when the user wants to benchmark on M-Cube, or asks about evaluating this task. Reports binary_evaluation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.22992
  bibtex_key: jiang2025marble
  confidence: high
---

# m-cube-eval

> MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning — Jiang et al. (2025) (arXiv:2506.22992, 2025)

## What this evaluates

Evaluates 3D spatial reasoning and combinatorial planning by requiring models to assemble jigsaw-style pieces into a 5x5x5 cube under physical constraints. The benchmark probes the model's ability to extract geometric patterns from rendered views and logically arrange pieces without gaps or overlaps.

## Datasets

- **M-Cube** — total 2000; splits: test (2000)

## Metrics

- `binary_evaluation` **(primary)** — range: percent
  - Binary success rate (0 or 1) indicating whether the model's proposed piece-to-face mapping and orientations successfully assemble into a perfect cube without gaps or overlaps, validated against geometric constraints and rotational symmetries.

## Input / output format

**Input**: Rendered 3D images of 6 jigsaw-style pieces from a random viewpoint (elevation -155° to -115°, azimuth -150° to -90°), or 2D arrays for the CUBE-easy variant. CUBE-easy also provides a partial solution with 4 pieces already placed.

**Output**: A piece-to-face assignment and orientation (rotation/flip) for each of the 6 pieces.

## Scoring recipe

```python
# For each of the 2000 instances
success_count = 0
for config in model_configs:
    if solution_validator(config):  # Checks edge alignment, gap/overlap, and rotational equivalence
        success_count += 1
accuracy = success_count / 2000
```

## Common pitfalls

- No unique ground-truth answer due to 24 rotational symmetries; exact string matching fails, requiring a geometric validator.
- Perception bottleneck: models struggle to extract bump/gap patterns from rendered images, causing immediate failure before reasoning begins.
- Combinatorial explosion: 6! piece assignments × 8 orientations per piece makes random guessing statistically negligible.

## Evidence (verbatim from paper)

> Therefore, instead of directly comparing the answer to ground-truth, we provide a solution validator by testing whether the solution from MLLM could successfully assemble the pieces into a perfect cube. Beside binary evaluation, the solution validator could also identify the conflicts in a given configuration, such as mismatched edges.

## Citation

```bibtex
@misc{jiang2025marble,
  title={MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2506.22992}
}
```

- arXiv: 2506.22992