Phuma Eval

---
name: phuma-eval
description: Evaluates a humanoid robot's ability to imitate human motion and follow pelvis trajectories using physically-grounded retargeting. It probes full-body tracking accuracy and partial-state path-following control across diverse locomotion categories on Unitree G1 and H1-2 robots. Use when the user wants to benchmark on PHUMA, Unseen Video, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.26236
  bibtex_key: lee2025phuma
  confidence: high
---

# phuma-eval

> PHUMA: Physically-Grounded Humanoid Locomotion Dataset — Kyungmin Lee et al. (arXiv:2510.26236, 2025)

## What this evaluates

Evaluates a humanoid robot's ability to imitate human motion and follow pelvis trajectories using physically-grounded retargeting. It probes full-body tracking accuracy and partial-state path-following control across diverse locomotion categories on Unitree G1 and H1-2 robots.

## Datasets

- **PHUMA** — total ?; splits: train (-1), test (7500)
- **Unseen Video** — total 504; splits: test (504)

## Metrics

- `success_rate` **(primary)** — range: percent
  - Ratio of motion sequences where the maximum deviation between the tracked trajectory and the target trajectory remains within a 0.15m threshold throughout the sequence.

## Input / output format

**Input**: Full-state: current proprioceptive state (joint positions, orientations, velocities) and full goal states (target motion trajectories). Partial-state: pelvis position and rotation.

**Output**: Joint angle commands executed via PD controllers.

## Scoring recipe

```python
def compute_success_rate(predictions, gold, threshold=0.15):
    successful = 0
    for motion_idx in range(len(predictions)):
        max_deviation = max(distance(predictions[motion_idx][t], gold[motion_idx][t]) for t in range(len(predictions[motion_idx])))
        if max_deviation <= threshold:
            successful += 1
    return (successful / len(predictions)) * 100
```

## Common pitfalls

- Using a 0.5m threshold incorrectly classifies scenarios as successful when humanoids remain stationary during jumps or stay upright during squatting motions.
- Failing to categorize motions into stationary, angular, vertical, and horizontal masks performance gaps in dynamic locomotion types.

## Evidence (verbatim from paper)

> For evaluating the full body motion tracking (RQ1, RQ2), we adopt the success rate metric from prior motion imitation studies (He et al., 2024b; 2025a; Xie et al., 2025), which measures the ratio of motions successfully imitated within a specified deviation threshold. Unlike prior work that uses a 0.5m threshold, we employ a stricter 0.15m threshold, as the standard threshold incorrectly classifies scenarios as successful when humanoids remain stationary during jumps or stay upright during squatting motions.

## Citation

```bibtex
@misc{lee2025phuma,
  title={PHUMA: Physically-Grounded Humanoid Locomotion Dataset},
  author={Kyungmin Lee et al.},
  year={2025},
  note={arXiv:2510.26236}
}
```

- arXiv: 2510.26236