Meta World Eval

---
name: meta-world-eval
description: Assesses multi-task robotic manipulation capabilities across varying difficulty levels (Easy, Medium, Hard, Very Hard) in simulation to evaluate robustness and generalization. Use when the user wants to benchmark on Meta-World, or asks about evaluating this task. Reports success rate (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.05684
  bibtex_key: choi2025d2e
  confidence: high
---

# meta-world-eval

> D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI — Suhwan Choi et al. (2025) (arXiv:2510.05684, 2025)

## What this evaluates

Assesses multi-task robotic manipulation capabilities across varying difficulty levels (Easy, Medium, Hard, Very Hard) in simulation to evaluate robustness and generalization.

## Datasets

- **Meta-World** — total ?; splits: test (-1)

## Metrics

- `success rate (%)` **(primary)** — range: percent
  - Percentage of successful rollouts out of total attempts per task category. Calculated as (number of successful episodes / total episodes) * 100.

## Input / output format

**Input**: Visual observations and task instructions.

**Output**: Robot action commands.

## Scoring recipe

```python
success_count = 0
for rollout in rollouts:
    if rollout.success:
        success_count += 1
success_rate = (success_count / len(rollouts)) * 100
```

## Common pitfalls

- Performance drops sharply on Hard/Very Hard tasks, so averaging across all difficulties can overstate general capability.
- Simulated physics parameters and task initializations can cause high variance in success rates.

## Evidence (verbatim from paper)

> Next, we evaluate our VAPT models on Meta-World, a standard benchmark for multi-task robotic manipulation... Even without robotics-specific pretraining or extensive hyperparameter tuning, VAPT consistently outperforms the baseline, showing an average success rate improvement of roughly 5% (a ~25% relative gain).

## Citation

```bibtex
@misc{choi2025d2e,
  title={D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI},
  author={Suhwan Choi et al. (2025)},
  year={2025},
  note={arXiv:2510.05684}
}
```

- arXiv: 2510.05684