Robot Manipulation Eval

---
name: robot-manipulation-eval
description: Evaluates how well vision foundation models support robot manipulation policies in simulation and real-world environments. It probes cross-modal spatial reasoning, task generalization across diverse manipulation suites, and robustness to sensor noise and platform differences. Use when the user wants to benchmark on LIBERO, MetaWorld, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.10105
  bibtex_key: [REDACTED]
  confidence: high
---

# robot-manipulation-eval

> EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation — Dong et al. (2025) (arXiv:2505.10105, 2025)

## What this evaluates

Evaluates how well vision foundation models support robot manipulation policies in simulation and real-world environments. It probes cross-modal spatial reasoning, task generalization across diverse manipulation suites, and robustness to sensor noise and platform differences.

## Datasets

- **LIBERO** — total 40; splits: test (40)
- **MetaWorld** — total 30; splits: test (30)

## Metrics

- `success rate` **(primary)** — range: percent
  - Calculated as the number of successful task completions divided by the total number of evaluation trials, multiplied by 100. Reported as an average across tasks or difficulty levels.

## Input / output format

**Input**: RGB and/or depth images (RGBD), point clouds, and robot proprioceptive state observations fed into a compact RDT policy network.

**Output**: Continuous action vectors for robot joint control.

## Scoring recipe

```python
def compute_success_rate(predictions, gold, trials_per_task=150):
    total_trials = len(gold) * trials_per_task
    successes = sum(1 for task in gold for i in range(trials_per_task) if predictions[task][i] == gold[task])
    return (successes / total_trials) * 100
```

## Common pitfalls

- Point cloud inputs often underperform RGB-only inputs in practice due to sensor noise from object reflectivity and lighting variations, contrary to prior assumptions about their compactness.
- Naively adding a trainable depth branch to RGB-only vision models (e.g., DINOv2) can degrade performance compared to using RGB-only inputs.

## Evidence (verbatim from paper)

> We report learning curves on LIBERO in [Figure 6] and success rate on MetaWorld in [Table 1]. Unless otherwise specified, “EmbodiedMAE” refers to the Large-scale, RGB-only variant. Each task is evaluated across 150 trials.

## Citation

```bibtex
@misc{dong2025embodiedmae,
  title={EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation},
  author={Dong et al. (2025)},
  year={2025},
  note={arXiv:2505.10105}
}
```

- arXiv: 2505.10105