Contact Rich Manipulation Eval

---
name: contact-rich-manipulation-eval
description: Evaluates the success rate of imitation learning policies in contact-rich manipulation tasks requiring precise force control, slip detection, and in-hand pose estimation. It compares vision-only baselines against visuo-tactile policies with and without temporal-aware contrastive pretraining. Use when the user wants to benchmark on Contact-Rich Manipulation Tasks, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.01941
  bibtex_key: wu2025freetacman
  confidence: high
---

# contact-rich-manipulation-eval

> FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation — Wu et al. (2025) (arXiv:2506.01941, 2025)

## What this evaluates

Evaluates the success rate of imitation learning policies in contact-rich manipulation tasks requiring precise force control, slip detection, and in-hand pose estimation. It compares vision-only baselines against visuo-tactile policies with and without temporal-aware contrastive pretraining.

## Datasets

- **Contact-Rich Manipulation Tasks** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Calculated as the number of successful task completions divided by the total number of evaluation trials (20 per task), expressed as a percentage. Success is determined by whether the task objective is met without damage or failure.
- `completion_rate` — range: percent
  - Fully completed tasks as a percentage of those initiated during the user study data collection phase.
- `CPUT` — range: other
  - Completion per Unit Time, defined as completion_rate multiplied by collection efficiency (inverse of data collection time).

## Input / output format

**Input**: RGB images from wrist camera (vision-only) or synchronized RGB images and tactile observations from wearable fingertip sensors (visuo-tactile).

**Output**: Robot joint control commands to execute the manipulation task.

## Scoring recipe

```python
def compute_success_rate(predictions, gold, total_trials=20):
    successes = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (successes / total_trials) * 100
```

## Common pitfalls

- The evaluation uses a fixed number of 20 trials per task rather than standard train/val/test splits.
- Confusing the data collection system metrics (CPUT, completion rate) with the policy evaluation metric (success rate).
- The tactile encoder pretraining uses a specific multi-positive contrastive objective, not standard supervised or self-supervised methods.

## Evidence (verbatim from paper)

> We collect data and train policies with tasks in Fig. 3 and evaluate each task over 20 trials. ... The vision-only baseline ACT achieves low performance across all tasks, with an average success rate of 21%. ... When tactile feedback is incorporated naively, i.e., without pre-training, performance improves significantly, with the average success rate increasing to 55%. ... Incorporating time-aware visual-tactile pairs in pretraining leads to a notable performance boost, with the average success rate increasing to 71%.

## Citation

```bibtex
@misc{wu2025freetacman,
  title={FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2506.01941}
}
```

- arXiv: 2506.01941