Humanoid Pose Control Eval

---
name: humanoid-pose-control-eval
description: Evaluates a language-conditioned transformer model's ability to generate physically plausible and text-aligned 3D humanoid poses from text commands. It probes motion quality, diversity, and multimodal alignment on a retargeted human motion benchmark, as well as real-world deployment success rates. Use when the user wants to benchmark on HumanoidML3D, Humanoid-X, or asks about evaluating this task. Reports FID.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.14172
  bibtex_key: mao2024humanoidx
  confidence: high
---

# humanoid-pose-control-eval

> Learning from Massive Human Videos for Universal Humanoid Pose Control — Jiageng Mao et al. (arXiv:2412.14172, 2024)

## What this evaluates

Evaluates a language-conditioned transformer model's ability to generate physically plausible and text-aligned 3D humanoid poses from text commands. It probes motion quality, diversity, and multimodal alignment on a retargeted human motion benchmark, as well as real-world deployment success rates.

## Datasets

- **HumanoidML3D** — total ?; splits: test (-1)
- **Humanoid-X** — total 20000000; splits: train (20000000)

## Metrics

- `FID` **(primary)** — range: other
  - Frechet Inception Distance measuring the dissimilarity between feature distributions of generated and ground-truth humanoid poses. Lower is better.
- `MM Dist` — range: other
  - Multi-modal Distance measuring the Euclidean distance between generated motions and corresponding text embeddings. Lower is better.
- `Diversity` — range: other
  - Average Euclidean distance between 300 randomly sampled pairs of generated humanoid poses. Higher is better.
- `R Precision` — range: [0, 1]
  - Accuracy of matching generated poses to their corresponding text descriptions within the top-3 ranked candidates. Higher is better.

## Input / output format

**Input**: Natural language text commands describing desired humanoid actions.

**Output**: 3D humanoid keypoint sequences or direct robotic joint action sequences.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, texts):
    fid = frechet_inception_distance(predictions.features, ground_truth.features)
    diversity = mean_pairwise_euclidean_distance(predictions, n_samples=300)
    mm_dist = mean_euclidean_distance(predictions.features, texts.embeddings)
    r_prec = top_k_accuracy(predictions.features, texts, k=3)
    return {'FID': fid, 'Diversity': diversity, 'MM Dist': mm_dist, 'R Precision': r_prec}
```

## Common pitfalls

- FID and MM Dist are highly sensitive to the choice of feature extractor and text encoder, making cross-paper comparisons difficult without identical backbones.
- Diversity is calculated over only 300 random pairs, which can introduce high variance if the generation distribution is multi-modal or sparse.
- Real-world success rates depend heavily on the control mode (open-loop vs. closed-loop) and the separation of upper/lower body control policies, which are not captured by offline metrics.

## Evidence (verbatim from paper)

> We adopt the metrics in [18] to evaluate the humanoid motions from different aspects: (1) Quality: The Frechet Inception Distance (FID) evaluates the dissimilarity between feature distributions of generated and ground truth humanoid poses. (2) Diversity: The Diversity metric evaluates the variability within the generated humanoid pose distribution, calculated as the average Euclidean distance between 300 randomly sampled pairs of humanoid poses. (3) Reliability: The Multi-modal Distance (MM Dist) measures the Euclidean distance between motions and corresponding texts, and the  $R$  Precision assesses the accuracy of text and humanoid pose matches in the Top 3 rankings.

## Citation

```bibtex
@misc{mao2024humanoidx,
  title={Learning from Massive Human Videos for Universal Humanoid Pose Control},
  author={Jiageng Mao et al.},
  year={2024},
  note={arXiv:2412.14172}
}
```

- arXiv: 2412.14172