Touchdown Eval

---
name: touchdown-eval
description: Evaluates an agent's capacity to navigate through street-view environments using natural language instructions and resolve complex spatial descriptions to locate a hidden target object within a panoramic image. Use when the user wants to benchmark on Touchdown, or asks about evaluating this task. Reports pixel distance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1811.12354
  bibtex_key: [REDACTED]
  confidence: medium
---

# touchdown-eval

> Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments — Chen et al. (2018) (arXiv:1811.12354, 2018)

## What this evaluates

Evaluates an agent's capacity to navigate through street-view environments using natural language instructions and resolve complex spatial descriptions to locate a hidden target object within a panoramic image.

## Datasets

- **Touchdown** — total 9326; splits: dev (-1), test (-1); repo https://github.com/lil-lab/touchdown

## Metrics

- `pixel distance` **(primary)** — range: pixels
  - Computed as the Euclidean distance in pixels between the predicted target location and the ground truth location. Accuracy is also reported as the fraction of predictions within an 80-pixel threshold.

## Input / output format

**Input**: A 3×84×84 RGB perspective image (projected from a 360° panorama centered on the agent's heading) paired with a natural language instruction describing navigation and spatial relations.

**Output**: A probability distribution over the image pixels, with the highest probability pixel indicating the predicted target location.

## Scoring recipe

```python
def score(predictions, golds):
    distances = [np.linalg.norm(p - g) for p, g in zip(predictions, golds)]
    acc = sum(1 for d in distances if d <= 80) / len(distances)
    return {'accuracy': acc, 'mean_pixel_distance': np.mean(distances)}
```

## Common pitfalls

- Models frequently latch onto visually salient or high-frequency objects (e.g., red signs, trash cans) rather than resolving precise spatial relations.
- Allocentric spatial reasoning is required, but models often fail when the scene contains dynamic changes or when the reference object is not uniquely identifiable by name alone.

## Evidence (verbatim from paper)

> The selected pixel is 104px from the correct one. ... the highest probability pixel is close enough (i.e., within 80 pixels) of the correct location.

## Citation

```bibtex
@misc{chen2018touchdown,
  title={Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments},
  author={Chen et al. (2018)},
  year={2018},
  note={arXiv:1811.12354}
}
```

- arXiv: 1811.12354