Peg Insertion Eval

---
name: peg-insertion-eval
description: Evaluates a robot policy's ability to perform contact-rich manipulation by jointly reasoning over visual and haptic feedback. It measures how well a learned representation improves sample efficiency, generalizes across peg geometries, and recovers from perturbations during peg insertion tasks. Use when the user wants to benchmark on Custom Peg Insertion Environment, or asks about evaluating this task. Reports sum of rewards achieved in an episode, normalized by the highest attainable reward.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.10191
  bibtex_key: lee2018makingsense
  confidence: high
---

# peg-insertion-eval

> Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks — Lee et al. (2018) (arXiv:1810.10191, 2018)

## What this evaluates

Evaluates a robot policy's ability to perform contact-rich manipulation by jointly reasoning over visual and haptic feedback. It measures how well a learned representation improves sample efficiency, generalizes across peg geometries, and recovers from perturbations during peg insertion tasks.

## Datasets

- **Custom Peg Insertion Environment** — total ?; splits: train (-1), eval (-1)

## Metrics

- `sum of rewards achieved in an episode, normalized by the highest attainable reward` **(primary)** — range: [0, 1]
  - Sum of rewards achieved in an episode divided by the highest attainable reward (10).
- `percentage of evaluation episodes` — range: percent
  - Percentage of evaluation episodes falling into four categories: completed insertion (peg reaches bottom), inserted into hole (peg enters but not bottom), touched the box (contact only), or failed (no contact).

## Input / output format

**Input**: 128x128 RGB image, end-effector pose, linear and angular velocity, and 6-axis force/torque vector.

**Output**: Action commands at 20Hz control frequency.

## Scoring recipe

```python
def compute_metrics(rewards_per_episode, episode_outcomes):
    max_reward = 10.0
    normalized_rewards = [sum(r) / max_reward for r in rewards_per_episode]
    stage_counts = {'completed': 0, 'inserted': 0, 'touched': 0, 'failed': 0}
    for outcome in episode_outcomes:
        stage_counts[outcome] += 1
    stage_rates = {k: v / len(episode_outcomes) * 100 for k, v in stage_counts.items()}
    return normalized_rewards, stage_rates
```

## Common pitfalls

- Confusing the internal staged reward phases (reaching, alignment) with the final evaluation categories (completed, inserted, touched, failed).
- Reporting raw cumulative reward instead of normalizing by the theoretical maximum of 10.
- Assuming simulation and real-robot sensor inputs are identical without accounting for Kinect vs. fixed camera and OptoForce vs. simulated contact models.

## Evidence (verbatim from paper)

> We report the quantitative performance of the policies using the sum of rewards achieved in an episode, normalized by the highest attainable reward. We also provide the statistics of the stages of the peg insertion task that each policy can achieve, and report the percentage of evaluation episodes in the following four categories: 1) completed insertion: the peg reaches bottom of the hole; 2) inserted into hole: the peg goes into the hole but has not reached the bottom; 3) touched the box: the peg only makes contact with the box; 4) failed: the peg fails to reach the box.

## Citation

```bibtex
@misc{lee2018makingsense,
  title={Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks},
  author={Lee et al. (2018)},
  year={2018},
  note={arXiv:1810.10191}
}
```

- arXiv: 1810.10191