Nuscenes Openloop Trajectory Eval

---
name: nuscenes-openloop-trajectory-eval
description: Assesses open-loop trajectory prediction accuracy of VLMs by measuring the distance between predicted and ground truth future paths at multiple time horizons. Use when the user wants to benchmark on nuScenes, or asks about evaluating this task. Reports L2 Error (m).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.23757
  bibtex_key: chi2025impromptu
  confidence: high
---

# nuscenes-openloop-trajectory-eval

> Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models — Chi et al. (2025) (arXiv:2505.23757, 2025)

## What this evaluates

Assesses open-loop trajectory prediction accuracy of VLMs by measuring the distance between predicted and ground truth future paths at multiple time horizons.

## Datasets

- **nuScenes** — total ?; splits: test (-1)

## Metrics

- `L2 Error (m)` **(primary)** — range: m
  - Euclidean distance in meters between predicted and ground truth trajectory points at 1s, 2s, and 3s horizons. Average L2 Error is computed across these horizons.

## Input / output format

**Input**: Historical trajectory and scene context from nuScenes.

**Output**: Predicted future trajectory points (x, y coordinates) for 1s, 2s, and 3s horizons.

## Scoring recipe

```python
def score_l2(predictions, gold):
    errors = []
    for horizon in [1, 2, 3]:
        pred_pts = predictions[horizon]
        gt_pts = gold[horizon]
        l2 = np.sqrt(np.sum((pred_pts - gt_pts)**2, axis=1))
        errors.append(np.mean(l2))
    return errors, np.mean(errors)
```

## Common pitfalls

- Evaluation is strictly open-loop; models do not receive feedback from the environment during prediction.
- Horizons are fixed at 1s, 2s, and 3s; average L2 is the mean across these three specific horizons.

## Evidence (verbatim from paper)

> Performance is primarily measured by the L2 distance (in meters) between the predicted and ground truth trajectories at future time horizons of 1s, 2s, and 3s, along with the average L2 error.

## Citation

```bibtex
@misc{chi2025impromptu,
  title={Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models},
  author={Chi et al. (2025)},
  year={2025},
  note={arXiv:2505.23757}
}
```

- arXiv: 2505.23757