Manipulation Transfer Eval

---
name: manipulation-transfer-eval
description: Evaluates whether learned hierarchical motor skills can transfer across different object geometries, downstream stacking tasks, and observation modalities (state vs. vision). It probes sample efficiency, directed exploration, and performance under varying reward sparsities (dense, staged sparse, fully sparse). Use when the user wants to benchmark on red_on_blue_stacking, all_pairs_stacking, or asks about evaluating this task. Reports reward.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.05062
  bibtex_key: rao2021learning
  confidence: high
---

# manipulation-transfer-eval

> Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies — Rao et al. (2021) (arXiv:2112.05062, 2021)

## What this evaluates

Evaluates whether learned hierarchical motor skills can transfer across different object geometries, downstream stacking tasks, and observation modalities (state vs. vision). It probes sample efficiency, directed exploration, and performance under varying reward sparsities (dense, staged sparse, fully sparse).

## Datasets

- **red_on_blue_stacking** — total ?; splits: (unstated)
- **all_pairs_stacking** — total ?; splits: (unstated)

## Metrics

- `reward` **(primary)** — range: other
  - Mean cumulative reward collected over evaluation episodes. Computed separately for dense, staged sparse, and fully sparse reward settings to measure task success and sample efficiency.
- `state_coverage_variance` — range: other
  - Variance of state subsets (joints, grasp sensor, object poses) over a single episode, averaged across episodes. Used to quantify the diversity of zero-shot exploration.

## Input / output format

**Input**: Environment state observations including proprioception (joint angles/velocities), object poses, and optionally RGB images for vision-based policies.

**Output**: Continuous action vector controlling the Sawyer arm's joints and gripper.

## Scoring recipe

```python
def compute_metrics(rollouts):
    rewards = [sum(r for r in episode) for episode in rollouts]
    avg_reward = sum(rewards) / len(rewards)
    coverage = []
    for episode in rollouts:
        states = [obs['state'] for obs in episode]
        coverage.append(np.var(states, axis=0))
    avg_coverage = np.mean(coverage, axis=0)
    return avg_reward, avg_coverage
```

## Common pitfalls

- Offline datasets are collected from a single source task but evaluated on geometrically distinct object sets and compositional tasks, which can mask adaptation failures.
- Vision-based transfer experiments still rely on privileged object states during training for KL-regularization, so zero-shot vision-only evaluation is not reported.
- Performance metrics vary significantly across reward sparsities; comparing dense vs. fully sparse results without noting the reward structure leads to misleading conclusions about sample efficiency.

## Evidence (verbatim from paper)

> We also record the total reward (dense and sparse staged). The results are reported in Table 1. While all approaches achieve some zero-shot dense reward (with BC the most effective), HeLMS receives a sparse staged reward an order of magnitude greater.

## Citation

```bibtex
@misc{rao2021learning,
  title={Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies},
  author={Rao et al. (2021)},
  year={2021},
  note={arXiv:2112.05062}
}
```

- arXiv: 2112.05062