Behavior1k Eval

---
name: behavior1k-eval
description: Evaluates embodied AI agents on long-horizon, human-centered manipulation tasks in a realistic physics-based simulation. It probes the agent's ability to plan and execute complex sequences of action primitives (pick, place, navigate, etc.) while handling rigid, articulated, and deformable objects. Use when the user wants to benchmark on BEHAVIOR-1K, or asks about evaluating this task. Reports task success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.09227
  bibtex_key: li2024behavior1k
  confidence: high
---

# behavior1k-eval

> BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation — Chengshu Li et al. (2024) (arXiv:2403.09227, 2024)

## What this evaluates

Evaluates embodied AI agents on long-horizon, human-centered manipulation tasks in a realistic physics-based simulation. It probes the agent's ability to plan and execute complex sequences of action primitives (pick, place, navigate, etc.) while handling rigid, articulated, and deformable objects.

## Datasets

- **BEHAVIOR-1K** — total 1000; splits: test (-1)

## Metrics

- `task success rate` **(primary)** — range: percent
  - Percentage of trials where the agent successfully completes the specified long-horizon manipulation task within the allowed horizon.
- `distance traveled` — range: other
  - Total Euclidean distance navigated by the robot base during the task, measured in meters.
- `simulated time` — range: other
  - Total simulation time elapsed until task completion or termination, measured in seconds.
- `kinematic object disarrangement` — range: other
  - Total displacement of objects caused by the robot's motion during the task, measured in meters.

## Input / output format

**Input**: RGB-D images, 3D object coordinates, and navigation/LiDAR data in real-world; corresponding OMNIGIBSON state observations and rendered images in simulation.

**Output**: Discrete selection of an action primitive (pick, place, push, navigate, dip, wipe) applied to a target object or location.

## Scoring recipe

```python
def compute_metrics(trials, gold_success):
    success_rate = sum(gold_success) / len(gold_success) * 100
    dist_nav = np.mean([t['dist_nav'] for t in trials])
    sim_time = np.mean([t['sim_time'] for t in trials])
    kin_dis = np.mean([t['kin_dis'] for t in trials])
    return success_rate, dist_nav, sim_time, kin_dis
```

## Common pitfalls

- Simplifying grasping via an 'assistive pick' primitive during training masks the true difficulty of physics-based grasping, causing severe performance drops when evaluated with full physics.
- Long-horizon tasks suffer from credit assignment and vanishing gradients in end-to-end visuomotor policies, requiring action primitives and memory to succeed.
- Sim-to-real transfer is heavily impacted by unmodeled visual discrepancies (camera dynamic range, textures) and navigation inaccuracies, not just policy errors.

## Evidence (verbatim from paper)

> Following the metrics proposed in BEHAVIOR-100 [27], we report the success rate and efficiency metrics (distance traveled, time invested, and disarrangement caused) in Table 2 and 3, and the success score Q in Table A.13 in Appendix.

## Citation

```bibtex
@misc{li2024behavior1k,
  title={BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation},
  author={Chengshu Li et al. (2024)},
  year={2024},
  note={arXiv:2403.09227}
}
```

- arXiv: 2403.09227