Vln Ce Eval

---
name: vln-ce-eval
description: Evaluates an agent's ability to follow natural language instructions to navigate to a target location in a continuous 3D environment. It probes low-level action control, obstacle avoidance, and spatial reasoning without relying on a pre-defined graph topology or oracle localization. Use when the user wants to benchmark on VLN-CE, or asks about evaluating this task. Reports SR, SPL.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.02857
  bibtex_key: krantz2020beyond
  confidence: high
---

# vln-ce-eval

> Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments — Krantz et al. (2020) (arXiv:2004.02857, 2020)

## What this evaluates

Evaluates an agent's ability to follow natural language instructions to navigate to a target location in a continuous 3D environment. It probes low-level action control, obstacle avoidance, and spatial reasoning without relying on a pre-defined graph topology or oracle localization.

## Datasets

- **VLN-CE** — total ?; splits: val-seen (-1), val-unseen (-1), test (-1); repo https://github.com/jacobkrantz/VLN-CE

## Metrics

- `TL` — range: meters
  - Total distance traveled by the agent along its trajectory.
- `NE` — range: meters
  - Euclidean distance between the agent's final position and the goal location at termination.
- `OS` — range: [0, 1]
  - Fraction of episodes where the agent reaches within 1 meter of the goal at any point during the trajectory.
- `SR` **(primary)** — range: [0, 1]
  - Fraction of episodes where the agent reaches within 1 meter of the goal at termination.
- `SPL` **(primary)** — range: [0, 1]
  - Success rate weighted by the inverse ratio of the agent's path length to the optimal path length.
- `nDTW` — range: [0, 1]
  - Normalized dynamic-time warping score measuring the similarity between the agent's trajectory and the ground truth path.

## Input / output format

**Input**: Egocentric RGB-D images, natural language navigation instructions, and history of previous actions/observations.

**Output**: Discrete low-level actions: forward, turn-left, turn-right, or stop.

## Scoring recipe

```python
def compute_spl(predictions, gold_paths):
    spl_scores = []
    for pred, gold in zip(predictions, gold_paths):
        final_pos = pred[-1]
        goal_pos = gold[-1]
        dist = euclidean_distance(final_pos, goal_pos)
        if dist <= 1.0:
            optimal_len = len(gold)
            actual_len = len(pred)
            spl_scores.append(optimal_len / actual_len)
        else:
            spl_scores.append(0.0)
    return sum(spl_scores) / len(predictions) if predictions else 0.0
```

## Common pitfalls

- Directly comparing VLN-CE performance to graph-based VLN benchmarks without accounting for the structural prior provided by the nav-graph.
- Assuming all VLN trajectories are navigable in continuous environments; ~20% are non-navigable and excluded in VLN-CE.

## Evidence (verbatim from paper)

> We report standard metrics for visual navigation tasks defined in [[4], [2], [18]] – trajectory length in meters (TL), navigation error in meters from goal at termination (NE), oracle success rate (OS), success rate (SR), success weighted by inverse path length (SPL), and normalized dynamic-time warping (nDTW). For our discussion, we will examine success rate and SPL as the primary metrics for performance and use NDTW to describe how paths differ in shape from ground truth trajectories.

## Citation

```bibtex
@misc{krantz2020beyond,
  title={Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments},
  author={Krantz et al. (2020)},
  year={2020},
  note={arXiv:2004.02857}
}
```

- arXiv: 2004.02857