Humanoid Everyday Eval

---
name: humanoid-everyday-eval
description: Evaluates imitation learning and vision-language-action policies on open-world humanoid manipulation. It probes robustness to high-dimensional action spaces, multimodal sensor fusion, and fine-grained visuospatial perception across locomotion, tool use, and precise manipulation tasks. Use when the user wants to benchmark on Humanoid Everyday, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.08807
  bibtex_key: zhao2025humanoid
  confidence: high
---

# humanoid-everyday-eval

> Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation — Zhao et al. (2025) (arXiv:2510.08807, 2025)

## What this evaluates

Evaluates imitation learning and vision-language-action policies on open-world humanoid manipulation. It probes robustness to high-dimensional action spaces, multimodal sensor fusion, and fine-grained visuospatial perception across locomotion, tool use, and precise manipulation tasks.

## Datasets

- **Humanoid Everyday** — total 10300; splits: train (-1), test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Calculated as the number of successful task completions divided by the total number of trials (10 per task), multiplied by 100.

## Input / output format

**Input**: Multimodal sensor observations (RGB, depth, LiDAR, tactile, IMU) and natural language task instructions, synchronized at 30Hz.

**Output**: 28-degree-of-freedom (DoF) robot action commands.

## Scoring recipe

```python
def compute_success_rate(successful_trials, total_trials=10):
    return (successful_trials / total_trials) * 100
# Evaluated over 10 trials per task. Average success rate is computed across all 7 tasks.
```

## Common pitfalls

- High-dimensional action space (28 DoFs) causes most end-to-end imitation policies to struggle significantly.
- Downsampling VLA model actions from 30Hz to 2Hz alleviates decoding failures but reduces motion smoothness.
- 3D point cloud inputs become unreliable in locomotion tasks due to large frame-to-frame camera changes.
- Fine-grained visuospatial perception is required for high-precision tasks, which current models largely lack.

## Evidence (verbatim from paper)

> We present the results of these policies in Table II. where each policy is evaluated over 10 trials on seven tasks. TABLE II: Success rates of imitation learning methods on the Humanoid Everyday Dataset.

## Citation

```bibtex
@misc{zhao2025humanoid,
  title={Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2510.08807}
}
```

- arXiv: 2510.08807