Habitat Benchmark Eval

---
name: habitat-benchmark-eval
description: Evaluates a robot's ability to perform long-horizon mobile manipulation and object rearrangement tasks in simulated environments. It probes hierarchical planning, whole-body continuous control, and robust recovery from failures across multi-step subtask sequences. Use when the user wants to benchmark on Habitat Benchmark, or asks about evaluating this task. Reports completion rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.17338
  bibtex_key: pezzato2025mobile
  confidence: high
---

# habitat-benchmark-eval

> Mobile Manipulation with Active Inference for Long-Horizon Rearrangement Tasks — Pezzato et al. (2025) (arXiv:2507.17338, 2025)

## What this evaluates

Evaluates a robot's ability to perform long-horizon mobile manipulation and object rearrangement tasks in simulated environments. It probes hierarchical planning, whole-body continuous control, and robust recovery from failures across multi-step subtask sequences.

## Datasets

- **Habitat Benchmark** — total ?; splits: test (-1)

## Metrics

- `completion rate` **(primary)** — range: percent
  - Percentage of episodes where the final subtask is successfully completed. A subtask failure cascades, meaning if any prior subtask fails, the entire episode is marked as failed regardless of later outcomes.

## Input / output format

**Input**: RGB-D images (128x128) from head and arm cameras, proprioceptive state (arm joint angles, end-effector Cartesian coordinates, goal positions, held-object scalar), and obstacle positions queried from the map model.

**Output**: 10-dimensional continuous action vector: forward and angular base velocities, 7DoF arm velocity, and 1DoF gripper action (positive for grasp, negative for release).

## Scoring recipe

```python
def compute_completion_rate(predictions, gold_subtasks):
    success_count = 0
    for pred in predictions:
        episode_failed = False
        for subtask in gold_subtasks:
            if not subtask.completed(pred.actions):
                episode_failed = True
                break
        if not episode_failed:
            success_count += 1
    return (success_count / len(predictions)) * 100
```

## Common pitfalls

- Subtask failures cascade: if any prior subtask fails, the entire episode is marked as failed regardless of subsequent subtask outcomes.
- Grasping is abstracted: a positive gripper action automatically snaps the nearest object within 15cm, bypassing real-world collision and precision constraints.
- Evaluation relies on privileged information (e.g., floor maps, articulated object states) that are queried externally rather than perceived by the policy.

## Evidence (verbatim from paper)

> we measure performance by reporting the completion rate at each subtask stage, with the success rate of the final subtask representing the overall task success. Notably, if the previous subtask has failed, the current subtask is also considered a failure independently of its outcome.

## Citation

```bibtex
@misc{pezzato2025mobile,
  title={Mobile Manipulation with Active Inference for Long-Horizon Rearrangement Tasks},
  author={Pezzato et al. (2025)},
  year={2025},
  note={arXiv:2507.17338}
}
```

- arXiv: 2507.17338