Tardis Stride Eval

---
name: tardis-stride-eval
description: Evaluates a generative world model's ability to produce controllable road images, predict geographic coordinates from street-view imagery, and generate self-consistent navigation actions on held-out spatiotemporal data. Use when the user wants to benchmark on STRIDE, or asks about evaluating this task. Reports georeferencing_error_m.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.11302
  bibtex_key: carriion2025tardis
  confidence: high
---

# tardis-stride-eval

> TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy — Carrión et al. (2025) (arXiv:2506.11302, 2025)

## What this evaluates

Evaluates a generative world model's ability to produce controllable road images, predict geographic coordinates from street-view imagery, and generate self-consistent navigation actions on held-out spatiotemporal data.

## Datasets

- **STRIDE** — total 3600000; splits: test (-1)

## Metrics

- `georeferencing_error_m` **(primary)** — range: meters
  - Euclidean or great-circle distance in meters between predicted coordinates and ground truth. Reported as a CDF of prediction errors.
- `road_adherence_rate` — range: percent
  - Percentage of generated navigation actions that result in a state within a specified distance threshold (e.g., 4m) of the nearest road centerline.
- `perplexity` — range: other
  - Standard cross-entropy loss exponentiated over generated tokens/images as a function of spatial and temporal commands.

## Input / output format

**Input**: Spatiotemporally held-out testing data consisting of panoramic road images, ground-truth states, and spatio-temporal commands (move distance/heading or temporal shifts).

**Output**: Generated images, predicted coordinate tokens (latitude/longitude), or generated navigation instructions (distance and heading).

## Scoring recipe

```python
def georeferencing_error(pred_latlon, true_latlon):
    return haversine_distance_m(pred_latlon, true_latlon)

def road_adherence_rate(actions, road_centerlines, threshold=4.0):
    valid = sum(1 for a in actions if distance_to_road(apply_action(a), road_centerlines) <= threshold)
    return (valid / len(actions)) * 100

def perplexity_metric(tokens):
    return exp(cross_entropy_loss(tokens))
```

## Common pitfalls

- Testing data is strictly the bottom 10% of the training area, held out both spatially and temporally.
- Georeferencing baseline (SVG) is artificially restricted to a 50m search space to match TARDIS's max move action, which may not reflect real-world open-set performance.
- Road adherence threshold varies (1m-10m); the paper reports 77.4% using a 4m threshold based on average US lane length, not a fixed standard.
- Perplexity is used as a proxy for image quality, which the authors explicitly note is often not representative of actual visual fidelity.

## Evidence (verbatim from paper)

> We measure all metrics and visualize all results using purely testing data which has been kept completely outside the models training regime spatiotemporally (bottom 10 percent of the training area). We measure quality quantitatively via the perplexity metric. We prompt TARDIS with 1…n testing nodes and measure its real-world meter error compared to the real coordinate. We measure this quantitatively by calculating the percent of actions which produce a following state within some distance threshold of the nearest road center line.

## Citation

```bibtex
@misc{carriion2025tardis,
  title={TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy},
  author={Carrión et al. (2025)},
  year={2025},
  note={arXiv:2506.11302}
}
```

- arXiv: 2506.11302