Personalized Embodied Navigation Eval

---
name: personalized-embodied-navigation-eval
description: Evaluates an embodied agent's ability to navigate and ground objects based on user-specific ownership semantics provided only in text. It tests long-term memory, spatial reasoning, and the capacity to interpret personalized queries without relying on visual object cues. Use when the user wants to benchmark on PersONAL, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.19843
  bibtex_key: ziliotto2025personal
  confidence: high
---

# personalized-embodied-navigation-eval

> PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agents — Ziliotto et al. (2025) (arXiv:2509.19843, 2025)

## What this evaluates

Evaluates an embodied agent's ability to navigate and ground objects based on user-specific ownership semantics provided only in text. It tests long-term memory, spatial reasoning, and the capacity to interpret personalized queries without relying on visual object cues.

## Datasets

- **PersONAL** — total 2000; splits: test (-1); repo https://github.com/ZiliottoFilippoDev/PersONAL

## Metrics

- `success_rate` **(primary)** — range: [0, 1]
  - Fraction of episodes where the agent selects the STOP action while positioned within 1.0 meters of the target object.

## Input / output format

**Input**: A personalized query (e.g., 'Find Lisa’s cabinet') and a detailed textual scene description specifying object ownership and spatial layout. For POG, these are provided after a pre-exploration phase.

**Output**: For PAN: a sequence of navigation actions (MOVE_FORWARD, TURN_LEFT/RIGHT, LOOK_UP/DOWN, STOP). For POG: coordinates on a pre-built spatial map.

## Scoring recipe

```python
def compute_success_rate(predictions, gold):
    successful = 0
    for pred, target in zip(predictions, gold):
        if pred['action'] == 'STOP' and distance(pred['pos'], target['pos']) <= 1.0:
            successful += 1
    return successful / len(predictions)
```

## Common pitfalls

- Confusing PAN (no prior knowledge, 500-step budget) with POG (allows 2,500-step pre-exploration and requires map coordinate output).
- Assuming image-based cues are used for ownership; the benchmark explicitly relies solely on textual descriptions.
- Ignoring strict step budget limits, which terminate episodes prematurely if exceeded.

## Evidence (verbatim from paper)

> An episode is considered successful if, after selecting the STOP action, the agent lies within 1.0 meters from the target object.

## Citation

```bibtex
@misc{ziliotto2025personal,
  title={PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agents},
  author={Ziliotto et al. (2025)},
  year={2025},
  note={arXiv:2509.19843}
}
```

- arXiv: 2509.19843