Nuplan Planning Eval

---
name: nuplan-planning-eval
description: Evaluates a driving planner's ability to navigate interactive scenarios in a closed-loop simulator. It measures success rates under both non-reactive (log-replay) and reactive (IDM/SMART) traffic conditions across routine validation splits and complex, human-curated scenarios. Use when the user wants to benchmark on Val14, Test14, interPlan, or asks about evaluating this task. Reports CLS-NR, CLS-R.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.03936
  bibtex_key: distelzweig2025drivingisagame
  confidence: high
---

# nuplan-planning-eval

> Driving is a Game: Combining Planning and Prediction with Bayesian Iterative Best Response — Distelzweig et al. (2025) (arXiv:2512.03936, 2025)

## What this evaluates

Evaluates a driving planner's ability to navigate interactive scenarios in a closed-loop simulator. It measures success rates under both non-reactive (log-replay) and reactive (IDM/SMART) traffic conditions across routine validation splits and complex, human-curated scenarios.

## Datasets

- **Val14** — total 1118; splits: val (1118)
- **Test14** — total ?; splits: random (-1), hard (280)
- **interPlan** — total 80; splits: full (80), lane-change (30)

## Metrics

- `CLS-NR` **(primary)** — range: percent
  - Closed-loop success score computed by running the planner in the nuPlan simulator over the full scenario horizon with non-reactive surrounding agents that follow pre-recorded trajectories.
- `CLS-R` **(primary)** — range: percent
  - Closed-loop success score computed under reactive surrounding agents using either the Intelligent Driver Model (IDM) or the SMART model to respond to ego behavior.

## Input / output format

**Input**: Historical agent trajectories $X_{1:N_a}^{-t_h:0}$ and map context $C_{\mathrm{map}}$.

**Output**: Ego trajectory $Y_{\mathrm{ego}}^t$ sampled at 10 Hz over a 4-second future horizon (40 positions).

## Scoring recipe

```python
def compute_nuplan_score(planner, scenario, agent_mode='NR'):
    sim = nuPlanSimulator(scenario, agent_mode=agent_mode)
    planner.run(sim)
    success_rate = sim.evaluate_closed_loop()
    return success_rate * 100
```

## Common pitfalls

- Confusing non-reactive (NR) and reactive (R) settings: NR replays logged trajectories, while R uses IDM or SMART models to react to the ego vehicle.
- Assuming Test14-hard is randomly sampled: it is curated by running PDM-Closed and selecting the 20 most challenging cases per category.
- Treating the metric as open-loop trajectory matching: evaluation is strictly closed-loop, meaning planning errors compound over the 4s horizon.

## Evidence (verbatim from paper)

> We report closed-loop non-reactive (CLS-NR) and closed-loop reactive (CLS-R) scores for Val14, Test14-hard, and Test14-random splits, as well as reactive scores on the interPlan and interPlanLC scenarios. Higher values indicate better performance.

## Citation

```bibtex
@misc{distelzweig2025drivingisagame,
  title={Driving is a Game: Combining Planning and Prediction with Bayesian Iterative Best Response},
  author={Distelzweig et al. (2025)},
  year={2025},
  note={arXiv:2512.03936}
}
```

- arXiv: 2512.03936