Roboverse Imitation Eval

---
name: roboverse-imitation-eval
description: Evaluates robot manipulation policies on a unified set of contact-rich pick-and-place and articulation tasks across multiple simulators. It probes both specialist and generalist vision-language-action models on success rates under standard and progressively challenging generalization levels. Use when the user wants to benchmark on ROBOVERSE Imitation Learning Benchmark, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.18904
  bibtex_key: geng2025roboverse
  confidence: high
---

# roboverse-imitation-eval

> RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning — Geng et al. (2025) (arXiv:2504.18904, 2025)

## What this evaluates

Evaluates robot manipulation policies on a unified set of contact-rich pick-and-place and articulation tasks across multiple simulators. It probes both specialist and generalist vision-language-action models on success rates under standard and progressively challenging generalization levels.

## Datasets

- **ROBOVERSE Imitation Learning Benchmark** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Binary success per episode averaged across task settings and random seeds. Reported as a percentage.

## Input / output format

**Input**: 256×256×3 RGB images and a short language description of the task.

**Output**: For specialist models: 9-dim robot joint state. For generalist models: delta end-effector position and binary gripper action {0, +1}, transformed to joint space via cuRobo IK solver.

## Scoring recipe

```python
total_success = 0.0
num_evals = len(task_settings) * 3
for setting in task_settings:
    for seed in [seed1, seed2, seed3]:
        success = run_episode(model, setting, seed)
        total_success += success
return (total_success / num_evals) * 100
```

## Common pitfalls

- Results are constrained to single-task fine-tuning due to resource limits, not multi-task or zero-shot generalization.
- OpenVLA baseline only evaluated on 20 uniformly sampled testing scenarios instead of the full set.
- Success metric definition may vary across source simulators (ManiSkill, RLBench, etc.), requiring careful alignment when comparing cross-benchmark results.

## Evidence (verbatim from paper)

> During evaluations, we randomly select ten task settings from training sets and another ten from the validation sets. The reported success rates are computed as the averages over three random seeds. For each step, the inputs are $256 	imes 256 	imes 3$ RGB images and a short language description depending on the task settings.

## Citation

```bibtex
@misc{geng2025roboverse,
  title={RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning},
  author={Geng et al. (2025)},
  year={2025},
  note={arXiv:2504.18904}
}
```

- arXiv: 2504.18904