One Shot Il Robot Manipulation Eval

---
name: one-shot-il-robot-manipulation-eval
description: This evaluation probes a robot's ability to generalize a single kinesthetic demonstration to novel object poses and orientations using unseen object pose estimation for trajectory transfer. It measures how robustly different pose estimation methods enable successful completion of everyday manipulation tasks in real-world settings. Use when the user wants to benchmark on Custom 10-task real-world manipulation set, or asks about evaluating this task. Reports success rate (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.12077
  bibtex_key: vitiello2023oneshot
  confidence: high
---

# one-shot-il-robot-manipulation-eval

> One-Shot Imitation Learning: A Pose Estimation Perspective — Vitiello et al. (2023) (arXiv:2310.12077, 2023)

## What this evaluates

This evaluation probes a robot's ability to generalize a single kinesthetic demonstration to novel object poses and orientations using unseen object pose estimation for trajectory transfer. It measures how robustly different pose estimation methods enable successful completion of everyday manipulation tasks in real-world settings.

## Datasets

- **Custom 10-task real-world manipulation set** — total 10; splits: test (10)

## Metrics

- `success rate (%)` **(primary)** — range: percent
  - Percentage of successful task executions out of 10 trials per method-task combination. A trial is successful if the robot completes the manipulation objective (e.g., placing, grasping, pouring) without failure.

## Input / output format

**Input**: Segmented RGB-D image of the scene, single kinesthetic demonstration trajectory (last-inch setting), and robot base state.

**Output**: End-effector trajectory / joint commands to execute the task from the estimated object pose.

## Scoring recipe

```python
def compute_success_rate(predictions, gold, num_trials=10):
    # predictions: list of 10 booleans (True if task completed successfully)
    # gold: represents task completion criteria
    successes = sum(1 for p in predictions if p)
    return (successes / num_trials) * 100
```

## Common pitfalls

- Confusing simulation pose estimation errors (cm/deg) with real-world task success rates; the former evaluates geometric accuracy, while the latter measures end-to-end manipulation success.
- Assuming a single trial determines performance; the protocol explicitly requires 10 trials per method-task combination to compute the success rate.
- Overlooking that DOME's failures stem largely from segmentation and visibility constraints rather than pose estimation errors alone.

## Evidence (verbatim from paper)

> The results for this experiment are shown in Table [2], with tasks ordered by mean success rate across methods and methods ordered by mean success rate across tasks. These results also include a comparison against DOME *[[20]]*, a state-of-the-art one-shot IL method.

## Citation

```bibtex
@misc{vitiello2023oneshot,
  title={One-Shot Imitation Learning: A Pose Estimation Perspective},
  author={Vitiello et al. (2023)},
  year={2023},
  note={arXiv:2310.12077}
}
```

- arXiv: 2310.12077