Hummi Wholebody Manipulation Eval

---
name: hummi-wholebody-manipulation-eval
description: Evaluates a humanoid robot's capability to learn and execute whole-body manipulation skills from robot-free demonstrations, assessing manipulation precision, dynamic coordination, generalization to unseen environments/objects, and data-collection efficiency. Use when the user wants to benchmark on Squatting, Tossing, Bimanual Actions, Walking, Dynamic Motions, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.06643
  bibtex_key: nai2026hummi
  confidence: low
---

# hummi-wholebody-manipulation-eval

> Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations — Nai et al. (2026) (arXiv:2602.06643, 2026)

## What this evaluates

Evaluates a humanoid robot's capability to learn and execute whole-body manipulation skills from robot-free demonstrations, assessing manipulation precision, dynamic coordination, generalization to unseen environments/objects, and data-collection efficiency.

## Datasets

- **Squatting** — total ?; splits: test (-1)
- **Tossing** — total ?; splits: test (-1)
- **Bimanual Actions** — total ?; splits: test (-1)
- **Walking** — total ?; splits: test (-1)
- **Dynamic Motions** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Percentage of trials where the robot successfully completes the manipulation task without failure, evaluated under in-domain and out-of-domain settings.
- `tracking error` — range: other
  - Mean Euclidean distance between the commanded trajectory and the executed trajectory, reported in centimeters.
- `data-collection efficiency` — range: other
  - Ratio of data points collected per unit time by HuMI compared to the baseline teleoperation system (TWIST2).

## Input / output format

**Input**: Full-body motion trajectories captured via handheld sensors and wearable trackers, including gripper, base, and foot positions/orientations.

**Output**: Robot execution of the learned whole-body manipulation policy; success/failure outcome and trajectory tracking metrics.

## Scoring recipe

```python
def score(predictions, gold):
    success_count = sum(1 for p in predictions if p == 'success')
    success_rate = (success_count / len(predictions)) * 100
    tracking_error = mean(sqrt(sum((cmd - exec)^2, axis=1)))
    efficiency = data_collected_hu / data_collected_twist
    return success_rate, tracking_error, efficiency
```

## Common pitfalls

- Evaluation covers both in-domain and out-of-domain settings, which significantly impacts reported success rates and generalization metrics.
- Data-collection efficiency is strictly compared against the TWIST2 teleoperation system, limiting direct comparison with other data collection methods.
- The exact threshold for 'successful' manipulation or acceptable tracking error is not specified in the provided text.

## Evidence (verbatim from paper)

> In this section, we empirically evaluate HuMI along three key dimensions. Specifically, we aim to answer the following questions: 1. Whole-body manipulation capability. Can HuMI learn feasible whole-body skills from robot-free demonstrations, achieve sufficient manipulation precision while respecting whole-body dynamics, and effectively coordinate high-low level policy during fully autonomous execution? 2. Generalization ability. Do robot-free demonstrations collected across varied environments enable the learned policy to generalize to unseen environments and objects? 3. Data-collection efficiency. Can HuMI acquire whole-body manipulation data efficiently and with a high acceptance rate, and does the collected dataset cover versatile whole-body skills, including motions that are challenging to obtain for teleoperation? To study these questions, we design five representative whole-body manipulation tasks and evaluate HuMI under both in-domain and out-of-domain settings with respect to environments and objects. To assess data-collection efficiency, we further compare HuMI against the state-of-the-art humanoid teleoperation system TWIST2.

## Citation

```bibtex
@misc{nai2026hummi,
  title={Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations},
  author={Nai et al. (2026)},
  year={2026},
  note={arXiv:2602.06643}
}
```

- arXiv: 2602.06643