Vln Task Planning Eval

---
name: vln-task-planning-eval
description: Evaluates an agent's ability to decompose coarse-grained natural language navigation instructions into executable subtasks and navigate through simulated environments to reach target locations or interact with objects. It probes task planning, visual-language grounding, and dynamic error recovery in continuous or discrete navigation spaces. Use when the user wants to benchmark on R2R, REVERIE, ALFRED, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.02535
  bibtex_key: zhao2024towards
  confidence: high
---

# vln-task-planning-eval

> Towards Coarse-grained Visual Language Navigation Task Planning Enhanced by Event Knowledge Graph — Zhao et al. (2024) (arXiv:2408.02535, 2024)

## What this evaluates

Evaluates an agent's ability to decompose coarse-grained natural language navigation instructions into executable subtasks and navigate through simulated environments to reach target locations or interact with objects. It probes task planning, visual-language grounding, and dynamic error recovery in continuous or discrete navigation spaces.

## Datasets

- **R2R** — total ?; splits: test (-1)
- **REVERIE** — total ?; splits: test (-1)
- **ALFRED** — total ?; splits: test (-1)

## Metrics

- `Success Rate (SR)` **(primary)** — range: [0, 1]
  - The ratio of episodes where the agent stops within 3 meters of the target location.
- `Success Path Length (SPL)` — range: [0, 1]
  - The success rate weighted by the normalized inverse of the path length (expert path length divided by agent path length).
- `Navigation Error (NE)` — range: meters
  - The average distance in meters between the agent's final position and the target.
- `Trajectory Length (TL)` — range: meters
  - The average path length in meters traversed by the agent.
- `Oracle Success Rate (OSR)` — range: [0, 1]
  - The ratio of trajectories that pass through at least one viewpoint where the target object is visible.
- `Conditional Success (GC)` — range: [0, 1]
  - The ratio of subtasks successfully completed over the entire task.
- `Path length weighted SR (PLWSR)` — range: [0, 1]
  - The ratio of the expert path length to the length taken by the agent.
- `Path length weighted GC (PLWGC)` — range: [0, 1]
  - The ratio of the ground truth path length to the length taken by the agent.

## Input / output format

**Input**: Coarse-grained natural language instruction and step-wise panoramic visual observations (local views). For ALFRED, interactive steps also require identifying target object masks.

**Output**: Discrete action selection: choose a navigable viewpoint to move to, or for ALFRED, select an action category (movement/interaction) and output a target object mask.

## Scoring recipe

```python
def compute_sr(final_pos, target_pos, threshold=3.0):
    return 1.0 if distance(final_pos, target_pos) <= threshold else 0.0

def compute_spl(final_pos, target_pos, agent_path_len, expert_path_len, threshold=3.0):
    sr = 1.0 if distance(final_pos, target_pos) <= threshold else 0.0
    if expert_path_len == 0: return 0.0
    return sr * (expert_path_len / agent_path_len)

def compute_ne(final_pos, target_pos):
    return distance(final_pos, target_pos)

def compute_tl(agent_path_len):
    return agent_path_len
```

## Common pitfalls

- The evaluation uses coarse-grained instructions without oracle fine-grained subtasks, so planners must handle ambiguous or high-level commands without ground-truth decomposition.
- The dynamic backtracking thresholds (x=0.25, W varies by dataset) are critical; setting them too low causes premature termination, while too high leads to step-limit failures.
- ALFRED requires joint action and object mask prediction, unlike R2R/REVERIE which only require viewpoint selection, complicating direct metric comparison across benchmarks.

## Evidence (verbatim from paper)

> For R2R: Trajectory Length (TL): the average path length in meters; Navigation Error (NE): the average distance between the agent’s final position and the target in meters; Success Rate (SR): the ratio of stopping within $3$ meters to the target; Success Path Length (SPL): the success rate weighted by the normalized inverse of the Path Length.

## Citation

```bibtex
@misc{zhao2024towards,
  title={Towards Coarse-grained Visual Language Navigation Task Planning Enhanced by Event Knowledge Graph},
  author={Zhao et al. (2024)},
  year={2024},
  note={arXiv:2408.02535}
}
```

- arXiv: 2408.02535