Humanoid Policy Eval

---
name: humanoid-policy-eval
description: Evaluates a unified state-action policy's ability to perform dexterous manipulation tasks on humanoid robots. It specifically probes in-distribution (I.D.) task execution and out-of-distribution (O.O.D.) generalization across varying backgrounds, object placements, and cross-embodiment transfers. Use when the user wants to benchmark on Robot & Human Manipulation Demonstrations, or asks about evaluating this task. Reports Success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.13441
  bibtex_key: qiuetal2025humanoidpolicy
  confidence: high
---

# humanoid-policy-eval

> Humanoid Policy ~ Human Policy — Ri-Zhao Qiu et al. (arXiv:2503.13441, 2025)

## What this evaluates

Evaluates a unified state-action policy's ability to perform dexterous manipulation tasks on humanoid robots. It specifically probes in-distribution (I.D.) task execution and out-of-distribution (O.O.D.) generalization across varying backgrounds, object placements, and cross-embodiment transfers.

## Datasets

- **Robot & Human Manipulation Demonstrations** — total ?; splits: train (-1), test (-1)

## Metrics

- `Success rate` **(primary)** — range: [0, 1]
  - Calculated as the number of successful autonomous skill executions divided by the total number of trials per task setting. Reported as a fraction (e.g., 42/60) or percentage.

## Input / output format

**Input**: Egocentric visual observations processed through a frozen DinoV2 ViT-S backbone, combined with proprioceptive state inputs (joint positions or unified state-action space representations).

**Output**: Future action chunks representing joint positions/actions for the humanoid robot's arms and hands.

## Scoring recipe

```python
def compute_success_rate(successes, total_trials):
    return successes / total_trials

# Per task setting (I.D. or O.O.D.):
# Count successful autonomous skill executions out of N trials
# Report fraction or percentage per task
```

## Common pitfalls

- Confusing I.D. and O.O.D. evaluation settings: I.D. closely matches training scene setups, while O.O.D. introduces novel backgrounds/object placements seen in human data but not robot data.
- Assuming co-training with human data improves in-distribution performance: the paper notes human data has minor effects on I.D. success, primarily boosting O.O.D. generalization.
- Ignoring action speed interpolation: baselines that do not slow down/interpolate human actions exhibit unstable action speeds and fail vertical grasping tasks.

## Evidence (verbatim from paper)

> Table 2: Success rate of autonomous skill execution. Co-training with human data (H. Data) significantly improves the Out-Of-Distribution (O.O.D.) performance with nearly 100% relative improvement on all tasks on Humanoid A.

## Citation

```bibtex
@misc{qiuetal2025humanoidpolicy,
  title={Humanoid Policy ~ Human Policy},
  author={Ri-Zhao Qiu et al.},
  year={2025},
  note={arXiv:2503.13441}
}
```

- arXiv: 2503.13441