Rxr Eval

---
name: rxr-eval
description: Evaluates the ability of embodied agents to follow natural language instructions for navigation in photo-realistic 3D environments. It probes multilingual understanding, spatial reasoning, and dense spatiotemporal grounding by measuring how accurately an agent navigates from a start to a target location. Use when the user wants to benchmark on Room-Across-Room (RxR), or asks about evaluating this task. Reports SR, NDTW.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.07954
  bibtex_key: ku2020roomacrossroom
  confidence: high
---

# rxr-eval

> Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding — Ku et al. (2020) (arXiv:2010.07954, 2020)

## What this evaluates

Evaluates the ability of embodied agents to follow natural language instructions for navigation in photo-realistic 3D environments. It probes multilingual understanding, spatial reasoning, and dense spatiotemporal grounding by measuring how accurately an agent navigates from a start to a target location.

## Datasets

- **Room-Across-Room (RxR)** — total 126000; splits: val-seen (-1), val-unseen (-1), test-standard (-1), test-challenge (-1); repo https://github.com/google-research-datasets/RxR

## Metrics

- `NE` — range: meters
  - Navigation Error: Euclidean distance (in meters) between the agent's final position and the target location. Lower is better.
- `SR` **(primary)** — range: [0, 1]
  - Success Rate: Fraction of trajectories that end within 3 meters of the target location. Higher is better.
- `NDTW` **(primary)** — range: [0, 1]
  - Normalized Dynamic Time Warping: Measures the similarity between the predicted path and the gold path, normalized by the gold path length. Higher is better.
- `SDTW` — range: [0, 1]
  - Soft Dynamic Time Warping: A differentiable approximation of DTW measuring path similarity. Higher is better.
- `SPL` — range: [0, 1]
  - Success weighted by Path Length: SR multiplied by the ratio of the gold path length to the predicted path length. Higher is better.

## Input / output format

**Input**: A sequence of panoramic images (36 views per step) with navigable direction encodings, paired with a natural language instruction in English, Hindi, or Telugu.

**Output**: A sequence of discrete navigation actions (e.g., move forward, turn left, turn right) until the agent stops at a target location or reaches a maximum step limit.

## Scoring recipe

```python
def compute_metrics(predicted_path, gold_path, target_pos):
    ne = euclidean_distance(predicted_path[-1], target_pos)
    sr = 1.0 if ne <= 3.0 else 0.0
    ndtw = normalized_dtw(predicted_path, gold_path)
    sdtw = soft_dtw(predicted_path, gold_path)
    spl = sr * (len(gold_path) / max(len(predicted_path), 1))
    return {'NE': ne, 'SR': sr, 'NDTW': ndtw, 'SDTW': sdtw, 'SPL': spl}
```

## Common pitfalls

- Val-unseen and test-standard splits are used for reporting, but test-challenge remains sequestered for a public leaderboard.
- Multilingual training on all three languages simultaneously degrades performance on high-resource English compared to monolingual training.
- Guide paths are algorithmically generated, while Follower paths are human demonstrations; mixing them changes the training distribution and evaluation dynamics.

## Evidence (verbatim from paper)

> Table 5 provides results on the val-unseen split for several training settings, as well as human performance from Follower annotations. ... NE ↓, SR ↑, SDTW ↑, NDTW ↑.

## Citation

```bibtex
@misc{ku2020roomacrossroom,
  title={Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding},
  author={Ku et al. (2020)},
  year={2020},
  note={arXiv:2010.07954}
}
```

- arXiv: 2010.07954