Goal Oriented Embodied Navigation Eval

---
name: goal-oriented-embodied-navigation-eval
description: Evaluates large multimodal models' ability to perform goal-oriented embodied navigation in complex urban 3D airspace. It probes geometric perception, cross-view understanding, spatial imagination, and long-term memory by requiring models to navigate from a start point to a semantic goal using visual observations and historical context. Use when the user wants to benchmark on Embodied Navigation Benchmark, or asks about evaluating this task. Reports SR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.07973
  bibtex_key: zhao2026embodiednavigationbench
  confidence: high
---

# goal-oriented-embodied-navigation-eval

> How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace — Zhao et al. (2026) (arXiv:2604.07973, 2026)

## What this evaluates

Evaluates large multimodal models' ability to perform goal-oriented embodied navigation in complex urban 3D airspace. It probes geometric perception, cross-view understanding, spatial imagination, and long-term memory by requiring models to navigate from a start point to a semantic goal using visual observations and historical context.

## Datasets

- **Embodied Navigation Benchmark** — total 5037; splits: short (-1), middle (-1), long (-1); repo https://github.com/serenditipy-AC/Embodied-Navigation-Bench

## Metrics

- `SR` **(primary)** — range: percent
  - Success Rate: the percentage of navigation trajectories that successfully reach the goal location within the allowed steps or distance threshold.
- `SPL` — range: [0, 1]
  - Success weighted by Path Length: SR multiplied by the ratio of the optimal path length to the actual path length taken. Measures both success and path efficiency.
- `accuracy` — range: percent
  - Reported as the percentage of correctly completed navigation tasks across different trajectory length groups (short, middle, long).

## Input / output format

**Input**: Current RGB observation combined with a memory buffer containing previous observations, actions, and rationales. For VLA baselines, inputs are aligned to match the goal-oriented navigation task format.

**Output**: Discrete spatial action commands expressed in natural language with an accompanying rationale, high-level reasoning/planning steps, or discrete action tokens, depending on the evaluation paradigm.

## Scoring recipe

```python
def evaluate_navigation(traj, goal, initial_pos, optimal_len):
    reached = distance(traj[-1], goal) < threshold
    progress = [distance(step, goal) / distance(initial_pos, goal) for step in traj]
    sr = 1.0 if reached else 0.0
    spl = sr * (1.0 if optimal_len == 0 else optimal_len / len(traj))
    return sr, spl

# Aggregate over dataset
SR = mean([sr for sr, _ in results]) * 100
SPL = mean([spl for _, spl in results])
```

## Common pitfalls

- Models are evaluated across three distinct paradigms (action-as-language, action-as-reasoning, action-as-token), so comparing raw scores across paradigms without accounting for output format differences is misleading.
- Navigation completion progress is non-linear; errors often trigger irreversible 'Critical Decision Bifurcations' (CDBs) where distance to goal increases monotonically, making standard linear error accumulation metrics insufficient.
- Trajectory length significantly impacts difficulty; short, middle, and long groups must be evaluated separately as performance gaps widen non-linearly with distance.

## Evidence (verbatim from paper)

> Both the random and action sample methods exhibit SR and SPL scores close to 0 in middle-distance and long-distance groups. This indicates that the task encompasses a vast action space.

## Citation

```bibtex
@misc{zhao2026embodiednavigationbench,
  title={How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2604.07973}
}
```

- arXiv: 2604.07973