Nuplan R Eval

---
name: nuplan-r-eval
description: Evaluates autonomous driving planners in a closed-loop setting with realistic, reactive multi-agent traffic. It probes a planner's ability to handle complex, interactive driving scenarios by measuring overall success, robustness against catastrophic failures, and consistency across safety and comfort dimensions. Use when the user wants to benchmark on nuPlan-R, or asks about evaluating this task. Reports CLS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.10403
  bibtex_key: peng2025nuplanr
  confidence: high
---

# nuplan-r-eval

> nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous Driving via Reactive Multi-Agent Simulation — Peng et al. (2025) (arXiv:2511.10403, 2025)

## What this evaluates

Evaluates autonomous driving planners in a closed-loop setting with realistic, reactive multi-agent traffic. It probes a planner's ability to handle complex, interactive driving scenarios by measuring overall success, robustness against catastrophic failures, and consistency across safety and comfort dimensions.

## Datasets

- **nuPlan-R** — total ?; splits: Test14-Random (-1), Test14-Hard (-1)

## Metrics

- `CLS` **(primary)** — range: percent
  - Standard Closed-Loop Success metric from the nuPlan benchmark, evaluating overall planner performance across safety, comfort, and rule compliance during closed-loop simulation.
- `SR` — range: percent
  - Success Rate, a binary or percentage metric indicating whether a planner reliably avoids catastrophic failures (e.g., collisions, severe violations) during closed-loop simulation.
- `PR` — range: percent
  - All-Core Pass Rate, measuring the fraction of core evaluation dimensions (e.g., safety, comfort, traffic rules) that a planner passes consistently across the scenario.

## Input / output format

**Input**: Scenario state including ego vehicle dynamics, surrounding traffic agents, and high-definition map features (e.g., vectorized polylines or rasterized BEV images).

**Output**: A planned trajectory for the ego vehicle over the simulation horizon, typically propagated via a kinematic bicycle model with LQR tracking.

## Scoring recipe

```python
def evaluate_nuplan_r(planner, scenario, reactive_agents):
    trajectory = run_closed_loop(planner, scenario, reactive_agents)
    cls = compute_cls(trajectory)
    sr = 1.0 if not has_catastrophic_failure(trajectory) else 0.0
    pr = compute_core_pass_rate(trajectory)
    return {'CLS': cls, 'SR': sr, 'PR': pr}
```

## Common pitfalls

- Using the original nuPlan's rule-based IDM agents instead of nuPlan-R's reactive diffusion-based agents, which artificially inflates rule-based planner scores.
- Relying exclusively on CLS, which can mask planner fragility or unbalanced behavior; SR and PR are required to detect catastrophic failure sensitivity and cross-dimensional consistency.
- Evaluating in open-loop mode; nuPlan-R requires frame-by-frame closed-loop simulation with reactive agent updates to capture multi-directional interactions.

## Evidence (verbatim from paper)

> In the original nuPlan benchmark, surrounding vehicles are controlled by a rule-based reactive agent based on the IDM, while in our nuPlan-R benchmark, they are driven by the proposed learning-based reactive agent, which enables more realistic interactions. In addition to the standard CLS metric, we also report two additional metrics, namely SR and PR metrics, as described in Section[III-C]. In all closed-loop simulations, we further employ a Linear Quadratic Regulator (LQR) controller for trajectory tracking, and the resulting control outputs are applied to a kinematic bicycle model to propagate the vehicle dynamics.

## Citation

```bibtex
@misc{peng2025nuplanr,
  title={nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous Driving via Reactive Multi-Agent Simulation},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2511.10403}
}
```

- arXiv: 2511.10403