Robobench Eval

---
name: robobench-eval
description: Evaluates multimodal large language models (MLLMs) as embodied brains in robotic manipulation. It probes five cognitive dimensions: instruction comprehension, perception reasoning, generalized planning, affordance prediction, and failure analysis across diverse real-world robotic tasks. Use when the user wants to benchmark on RoboBench, or asks about evaluating this task. Reports accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.17801
  bibtex_key: luo2025robobench
  confidence: high
---

# robobench-eval

> Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain — Luo et al. (2025) (arXiv:2510.17801, 2025)

## What this evaluates

Evaluates multimodal large language models (MLLMs) as embodied brains in robotic manipulation. It probes five cognitive dimensions: instruction comprehension, perception reasoning, generalized planning, affordance prediction, and failure analysis across diverse real-world robotic tasks.

## Datasets

- **RoboBench** — total 6092; splits: test (6092)

## Metrics

- `accuracy (%)` **(primary)** — range: percent
  - Calculated as the percentage of correctly answered questions out of the total number of questions per dimension. Scores are reported for each of the five cognitive dimensions and averaged for an overall score.

## Input / output format

**Input**: Multimodal inputs consisting of real-world robotic scene images or videos paired with natural language instructions or questions.

**Output**: Text-based answers to the provided questions, evaluated for correctness against ground-truth labels.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

# Report per dimension and average across the five dimensions
```

## Common pitfalls

- Models show a ~30% performance drop on implicit instructions compared to explicit ones, indicating weak intent grounding.
- Spatiotemporal reasoning and causal grounding across time are consistently weak, with temporal grounding accuracy often below 50%.
- Cross-embodiment planning (e.g., dual-arm coordination, mobile manipulation) remains a major bottleneck for current MLLMs.

## Evidence (verbatim from paper)

> Perception Challenges: While models exhibit reasonable performance in object property analysis, they struggle with fundamental robotic perception and spatiotemporal reasoning. Common failures include misidentifying the robot type or viewpoint, and being unable to accurately localize events in time. In Table 3, the success rate on robotic-view recognition is only 43.48, and temporal grounding accuracy reaches merely 49.68.

## Citation

```bibtex
@misc{luo2025robobench,
  title={Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2510.17801}
}
```

- arXiv: 2510.17801