Maniskill2 Eval

---
name: maniskill2-eval
description: Evaluates the generalization and robustness of embodied AI manipulation policies across soft-body, rigid-body, and assembly tasks in a simulated environment. Use when the user wants to benchmark on ManiSkill2, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.11343
  bibtex_key: gao2023twostage
  confidence: medium
---

# maniskill2-eval

> A Two-stage Fine-tuning Strategy for Generalizable Manipulation Skill of Embodied AI — Fang Gao et al. (2023) (arXiv:2307.11343, 2023)

## What this evaluates

Evaluates the generalization and robustness of embodied AI manipulation policies across soft-body, rigid-body, and assembly tasks in a simulated environment.

## Datasets

- **ManiSkill2** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: [0, 1]
  - The proportion of simulation episodes in which the robot successfully completes the specified manipulation task.

## Input / output format

**Input**: Visual observations (RGB/depth) and proprioceptive states from the ManiSkill2 simulation environment.

**Output**: Continuous action commands (e.g., joint velocities or end-effector poses) for the robotic manipulator.

## Scoring recipe

```python
def compute_success_rate(rollouts):
    total = len(rollouts)
    successes = sum(1 for r in rollouts if r['success'] == 1)
    return successes / total
```

## Common pitfalls

- Overfitting to specific task dynamics during the first-stage fine-tuning.
- Sensitivity to hyperparameter scaling factors for batch size and sample density during the second stage.

## Evidence (verbatim from paper)

> The results of these experiments, as shown in Table I, indicate that the scale of 0.9 for the batch size and 0.875 for the number of samples yielded better performance. To further illustrate the effectiveness of our two-stage fine-tuning strategy, we visualized the trend line of the success rate with and without our strategy in Fig. 2.

## Citation

```bibtex
@misc{gao2023twostage,
  title={A Two-stage Fine-tuning Strategy for Generalizable Manipulation Skill of Embodied AI},
  author={Fang Gao et al. (2023)},
  year={2023},
  note={arXiv:2307.11343}
}
```

- arXiv: 2307.11343