Robotic Manipulation Eval

---
name: robotic-manipulation-eval
description: Evaluates the ability of diffusion transformer policies to perform long-horizon robotic manipulation tasks across bi-manual, single-arm, and simulated environments. It probes stable training, observation tokenization, and generalization across different robot morphologies and action spaces. Use when the user wants to benchmark on Robotic Manipulation Task Suite, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.10088
  bibtex_key: [REDACTED]
  confidence: high
---

# robotic-manipulation-eval

> The Ingredients for Robotic Diffusion Transformers — Dasari et al. (2024) (arXiv:2410.10088, 2024)

## What this evaluates

Evaluates the ability of diffusion transformer policies to perform long-horizon robotic manipulation tasks across bi-manual, single-arm, and simulated environments. It probes stable training, observation tokenization, and generalization across different robot morphologies and action spaces.

## Datasets

- **Robotic Manipulation Task Suite** — total ?; splits: test (-1)

## Metrics

- `success_rate` **(primary)** — range: percent
  - Percentage of episodes where the robot successfully completes the specified manipulation task within the allotted time/steps.

## Input / output format

**Input**: Multi-view camera images, goal conditioning vectors, and language instructions.

**Output**: Continuous robot action vectors (e.g., joint velocities or end-effector poses).

## Scoring recipe

```python
def compute_success_rate(predictions, gold, task_config):
    success_count = 0
    for pred, gold_step in zip(predictions, gold):
        if check_task_completion(pred, task_config):
            success_count += 1
    return (success_count / len(gold)) * 100
```

## Common pitfalls

- Baselines like D.P. Transformer often fail to train stably without extensive hyperparameter tuning, leading to unsafe/noisy action predictions.
- Sim-to-real evaluation gap means high simulation success rates do not guarantee real-world performance.

## Evidence (verbatim from paper)

> We evaluate DiT-Block Policy on a set of 3 Bi-Manual and 2 Single-Arm manipulation tasks... We find that DiT-Block Policy is able to outperform the strongest by roughly 20% when trained with BiPlay, and by 10% when trained on task data alone.

## Citation

```bibtex
@misc{dasari2024ingredients,
  title={The Ingredients for Robotic Diffusion Transformers},
  author={Dasari et al. (2024)},
  year={2024},
  note={arXiv:2410.10088}
}
```

- arXiv: 2410.10088