Molmospaces Bench Eval

---
name: molmospaces-bench-eval
description: Evaluates zero-shot generalization of vision-language-action and navigation policies across diverse indoor scenes. Probes robustness to environmental perturbations, language prompt variations, and sim-to-real transferability for long-horizon manipulation and semantic navigation tasks. Use when the user wants to benchmark on MolmoSpaces-Bench, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.11337
  bibtex_key: [REDACTED]
  confidence: high
---

# molmospaces-bench-eval

> MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation — Kim et al. (2026) (arXiv:2602.11337, 2026)

## What this evaluates

Evaluates zero-shot generalization of vision-language-action and navigation policies across diverse indoor scenes. Probes robustness to environmental perturbations, language prompt variations, and sim-to-real transferability for long-horizon manipulation and semantic navigation tasks.

## Datasets

- **MolmoSpaces-Bench** — total ?; splits: navigate-to (2000), manipulation (-1); repo https://github.com/allenai/molmospaces

## Metrics

- `success rate` **(primary)** — range: [0, 1]
  - Fraction of episodes where the robot successfully completes the specified task within the allowed action horizon.
- `Pearson correlation coefficient` — range: [-1, 1]
  - Linear correlation between simulation success rates and real-world success rates across tasks to measure predictive validity.
- `Spearman rank correlation coefficient` — range: [-1, 1]
  - Rank-based correlation between simulation and real-world success rates to assess monotonic relationship and robustness to outliers.

## Input / output format

**Input**: RGB-D visual observations (from wrist and third-person cameras) paired with natural language instructions (e.g., semantic navigation commands like 'go to [object]' or manipulation prompts).

**Output**: Robot action commands (joint positions, gripper states) or binary task completion status.

## Scoring recipe

```python
def compute_metrics(predictions, gold, sim_rates, real_rates):
    # Success rate
    successes = sum(1 for p, g in zip(predictions, gold) if p == g)
    success_rate = successes / len(gold)
    
    # Correlations (sim vs real)
    pearson_r = pearsonr(sim_rates, real_rates)
    spearman_rho = spearmanr(sim_rates, real_rates)
    
    return success_rate, pearson_r, spearman_rho
```

## Common pitfalls

- Task horizon must be set to 300 for π models and 50 for CAP to avoid premature termination bias and misrepresenting retry behavior.
- Prompt phrasing significantly impacts performance due to training data distribution shifts; using non-DROID frequent verbs degrades results by up to 14%.
- Camera occlusion drastically reduces success rates, especially for wrist-mounted cameras (drops to 2%), which must be accounted for in robustness evaluations.

## Evidence (verbatim from paper)

> We evaluate the correlation for the pick, open, and close tasks individually. Results for are shown in [Fig. 11]. For the pick task, we observe a strong linear correlation between our MolmoSpaces-Bench results and the results from 752 RoboArena pick tasks, with Pearson and Spearman rank correlation coefficients of 0.96 and 0.98, respectively.

## Citation

```bibtex
@misc{kim2026molmospaces,
  title={MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation},
  author={Kim et al. (2026)},
  year={2026},
  note={arXiv:2602.11337}
}
```

- arXiv: 2602.11337