Maniskill2 Softbody Eval

---
name: maniskill2-softbody-eval
description: Evaluates a robot policy's ability to perform long-horizon manipulation tasks involving deformable soft bodies (e.g., clay, noodles, liquid, plasticine). It probes spatial reasoning, contact dynamics, and precise end-effector control under varying initial conditions. Use when the user wants to benchmark on ManiSkill2 Challenge (Soft-body Track), or asks about evaluating this task. Reports Success Metric.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.05756
  bibtex_key: li2024learninggeneralizable
  confidence: high
---

# maniskill2-softbody-eval

> Learning the Generalizable Manipulation Skills on Soft-body Tasks via Guided Self-attention Behavior Cloning Policy — Li et al. (2024) (arXiv:2410.05756, 2024)

## What this evaluates

Evaluates a robot policy's ability to perform long-horizon manipulation tasks involving deformable soft bodies (e.g., clay, noodles, liquid, plasticine). It probes spatial reasoning, contact dynamics, and precise end-effector control under varying initial conditions.

## Datasets

- **ManiSkill2 Challenge (Soft-body Track)** — total ?; splits: test (450)

## Metrics

- `Success Metric` **(primary)** — range: [0, 1]
  - Fraction of successful episodes out of total evaluation episodes per task. Success is determined by task-specific criteria: volume thresholds (Fill), geometric constraints (Hang, Excavate, Pour), Chamfer distance (Pinch), or IoU (Write), plus velocity limits (<0.05) where specified.

## Input / output format

**Input**: Point cloud observations and end-effector state in the robot's frame, processed via a guided self-attention module for long-horizon interaction modeling.

**Output**: End-effector delta position commands via a pd-joint-delta-pos controller.

## Scoring recipe

```python
def compute_success_rate(task, episodes):
    successes = 0
    for ep in episodes:
        if task == 'Fill':
            ok = (ep.volume_in_beaker > 0.9 * ep.capacity) and (ep.velocity < 0.05)
        elif task == 'Hang':
            ok = (ep.noodle_above_rod) and (ep.ends_on_opposite_sides) and (not ep.touches_ground) and (ep.gripper_open) and (ep.velocity < 0.05)
        elif task == 'Excavate':
            ok = (ep.lifted_volume >= ep.threshold) and (ep.height >= ep.height_threshold) and (ep.spillage < 20) and (ep.velocity < 0.05)
        elif task == 'Pour':
            ok = (abs(ep.liquid_level - ep.target_level) <= 0.004) and (ep.spillage < 100) and (ep.bottle_upright) and (ep.arm_velocity < 0.05)
        elif task == 'Pinch':
            ok = (ep.chamfer_dist < 0.3 * ep.initial_chamfer_dist)
        elif task == 'Write':
            ok = (ep.iou > 0.8)
        else:
            ok = False
        if ok: successes += 1
    return successes / len(episodes)
```

## Common pitfalls

- Task-specific success criteria vary widely (e.g., Chamfer distance vs. IoU vs. volume thresholds), requiring separate validation logic per task.
- Velocity constraints (<0.05) are mandatory for success in Fill, Hang, Excavate, and Pour, but omitted for Pinch and Write; ignoring them causes false positives.
- Evaluation uses randomized initial conditions (poses, target shapes/characters), making results highly sensitive to the random seed and episode count.

## Evidence (verbatim from paper)

> ManiSkill2 challenge includes 6 soft-body manipulation tasks that call for agents to engage with soft bodies (refer to Fig. 4), moving or deforming them to achieve predetermined target states. Success Metric: The task is successful when the volume of clay inside the target beaker exceeds 90% of its capacity, while maintaining the soft body velocity below 0.05. Evaluation Protocol: Conduct 100 episodes with varying initial rotations of the bucket and initial positions of the beaker.

## Citation

```bibtex
@misc{li2024learninggeneralizable,
  title={Learning the Generalizable Manipulation Skills on Soft-body Tasks via Guided Self-attention Behavior Cloning Policy},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2410.05756}
}
```

- arXiv: 2410.05756