Bsc Nav Eval

---
name: bsc-nav-eval
description: Evaluates embodied agents' spatial cognition and navigation capabilities across category-level, instance-level, and long-horizon instruction-following tasks, as well as active embodied question answering and real-world mobile manipulation. Use when the user wants to benchmark on MP3D & HM3D (Habitat Simulator), VLN-CE R2R, OpenEQA (A-EQA subset), Real-world Indoor Environment, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.17198
  bibtex_key: ruan2025bscnav
  confidence: high
---

# bsc-nav-eval

> From reactive to cognitive: brain-inspired spatial intelligence for embodied agents — Ruan et al. (2025) (arXiv:2508.17198, 2025)

## What this evaluates

Evaluates embodied agents' spatial cognition and navigation capabilities across category-level, instance-level, and long-horizon instruction-following tasks, as well as active embodied question answering and real-world mobile manipulation.

## Datasets

- **MP3D & HM3D (Habitat Simulator)** — total 8195; splits: seen (-1), unseen (-1)
- **VLN-CE R2R** — total 1000; splits: test (-1)
- **OpenEQA (A-EQA subset)** — total 184; splits: test (-1)
- **Real-world Indoor Environment** — total 75; splits: test (-1)

## Metrics

- `Success Rate (SR)` **(primary)** — range: [0, 1]
  - Binary metric: 1 if the agent executes the 'stop' action within 1.0 m of the goal location, 0 otherwise. Averaged over all episodes.
- `Success weighted by Path Length (SPL)` — range: [0, 1]
  - Standard SPL metric: SR multiplied by the ratio of the shortest possible path length to the actual path length taken by the agent. Averages efficiency and success.
- `LLM-Match` — range: other
  - Semantic similarity score between the agent's generated answer and the reference answer, computed using an LLM-based evaluator.

## Input / output format

**Input**: Egocentric visual observations, text or image navigation instructions, and spatial context from structured memory modules.

**Output**: Discrete navigation actions (forward 25 cm, turn left 30°, turn right 30°, stop) or natural language answers for QA tasks.

## Scoring recipe

```python
def compute_sr(predictions, gold):
    success = 0
    for pred, goal in zip(predictions, gold):
        dist = calculate_distance(pred.stop_pos, goal.pos)
        if dist <= 1.0:
            success += 1
    return success / len(predictions)

def compute_spl(predictions, gold):
    sr = compute_sr(predictions, gold)
    path_ratios = []
    for pred, goal in zip(predictions, gold):
        shortest = goal.shortest_path_len
        actual = pred.actual_path_len
        if shortest > 0:
            path_ratios.append(actual / shortest)
        else:
            path_ratios.append(0)
    return sr * (sum(path_ratios) / len(path_ratios))
```

## Common pitfalls

- Success is strictly defined as stopping within 1.0 m of the goal, not just reaching the vicinity.
- LIN/VLN-CE evaluation is zero-shot; do not assume instruction-level supervised training.
- LLM-Match requires careful prompt design for semantic similarity, as it replaces exact string matching.

## Evidence (verbatim from paper)

> Following standard protocols, navigation succeeds only when agents execute stop within 1.0 m of goals. Evaluation metrics include the Success Rate (SR) for efficacy and the Success weighted by Path Length (SPL) for efficiency (detailed in Methods).

## Citation

```bibtex
@misc{ruan2025bscnav,
  title={From reactive to cognitive: brain-inspired spatial intelligence for embodied agents},
  author={Ruan et al. (2025)},
  year={2025},
  note={arXiv:2508.17198}
}
```

- arXiv: 2508.17198