Lead Closed Loop Eval

---
name: lead-closed-loop-eval
description: Evaluates the closed-loop driving performance and generalization of end-to-end autonomous driving policies in simulation and on real-world datasets. It probes the model's ability to navigate long-horizon routes, handle diverse weather and lighting conditions, and transfer synthetic pre-training to real-world driving scenarios without violating traffic rules. Use when the user wants to benchmark on CARLA Town13, Bench2Drive, Longest6 v2, NAVSIM v1, NAVSIM v2, WOD-E2E, or asks about evaluating this task. Reports NDS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.20563
  bibtex_key: nguyen2025lead
  confidence: high
---

# lead-closed-loop-eval

> LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving — Long Nguyen et al. (arXiv:2512.20563, 2025)

## What this evaluates

Evaluates the closed-loop driving performance and generalization of end-to-end autonomous driving policies in simulation and on real-world datasets. It probes the model's ability to navigate long-horizon routes, handle diverse weather and lighting conditions, and transfer synthetic pre-training to real-world driving scenarios without violating traffic rules.

## Datasets

- **CARLA Town13** — total ?; splits: val (-1), train (-1)
- **Bench2Drive** — total ?; splits: test (-1)
- **Longest6 v2** — total ?; splits: test (-1)
- **NAVSIM v1** — total ?; splits: navtrain (-1), navtest (-1)
- **NAVSIM v2** — total ?; splits: navhard (-1)
- **WOD-E2E** — total ?; splits: validation (-1)

## Metrics

- `NDS` **(primary)** — range: [0, 1]
  - Normalized Driving Score, defined as RC × I, where RC is Route Completion and I is the distance-normalized Infraction Score.
- `Driving Score (DS)` — range: [0, 1]
  - Standard CARLA metric combining route completion and infraction penalties. Can be counterintuitive on long routes due to exponential IS decay.
- `Success Rate (SR)` — range: [0, 1]
  - The fraction of routes completed with zero infractions.
- `PDMS` — range: [0, 1]
  - Predictive Driver Model Score aggregating collision avoidance, progress, time-to-collision, driving-area compliance, and comfort.
- `EPDMS` — range: [0, 1]
  - Extended PDMS used in NAVSIM v2, incorporating additional rule-based and sub-metrics with proximity-based weighting for synthetic start states.
- `RFS` — range: [0, 1]
  - Rater Feedback Score, a human-annotated metric assigning full expert credit within predefined trust regions and exponentially decaying the score otherwise.

## Input / output format

**Input**: Multi-view camera images, LiDAR point clouds, and radar object-level features (or positional encodings when sensors are unavailable), along with historical vehicle states and discrete navigation commands.

**Output**: Continuous driving control commands (steering, throttle, brake) or a planned trajectory for the next time step.

## Scoring recipe

```python
def compute_nds(route_completion, infraction_score, distance):
    I = infraction_score / distance if distance > 0 else 0
    return route_completion * I

def compute_success_rate(routes):
    infraction_free = sum(1 for r in routes if r.infractions == 0)
    return infraction_free / len(routes)

def compute_pdms(trajectory, ground_truth, environment):
    # Aggregates collision avoidance, progress, TTC, drivable area compliance, comfort
    return weighted_sum([collision_avoidance, progress, ttc, area_compliance, comfort])
```

## Common pitfalls

- Driving Score (DS) can be misleading on long routes because the exponential decay of Infraction Score (IS) may reward agents for stopping early rather than completing the route.
- Training on data from the validation town (e.g., Town13 Train) artificially inflates performance and hides the true generalization gap to novel environments.
- DS discounts errors on short routes, making Success Rate (SR) necessary to accurately assess policy robustness and infraction-free completion.

## Evidence (verbatim from paper)

> Metrics: We adopt the standard CARLA Leaderboard 2.0 metrics from Section 3.1. On long routes, however, the Driving Score (DS) can be counterintuitive: because the Infraction Score (IS) decays exponentially with each violation, agents are sometimes rewarded for stopping early rather than driving further [64]. We therefore report the Normalized Driving Score (NDS) on Town13, defined as RC × I, where I is distance-normalized version of IS [64]. For Bench2Drive, we additionally report Success Rate (SR), the fraction of infraction-free completions.

## Citation

```bibtex
@misc{nguyen2025lead,
  title={LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving},
  author={Long Nguyen et al.},
  year={2025},
  note={arXiv:2512.20563}
}
```

- arXiv: 2512.20563