Navsim Pdm Eval

---
name: navsim-pdm-eval
description: Evaluates end-to-end autonomous driving planners on trajectory prediction and safety-critical behaviors. It measures compliance with traffic rules, drivable area boundaries, collision avoidance, and driving comfort over short-horizon scenarios. Use when the user wants to benchmark on NAVSIM, or asks about evaluating this task. Reports PDM Score (PDMS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.12820
  bibtex_key: li2025hydramdpp
  confidence: high
---

# navsim-pdm-eval

> Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillation — Li et al. (2025) (arXiv:2503.12820, 2025)

## What this evaluates

Evaluates end-to-end autonomous driving planners on trajectory prediction and safety-critical behaviors. It measures compliance with traffic rules, drivable area boundaries, collision avoidance, and driving comfort over short-horizon scenarios.

## Datasets

- **NAVSIM** — total 1328; splits: train (1192), test (136)

## Metrics

- `PDM Score (PDMS)` **(primary)** — range: percent
  - PDMS = NC × DAC × (5×TTC + 2×C + 5×EP) / 12. Sub-metrics are percentages: No at-fault Collision (NC), Drivable Area Compliance (DAC), Time-to-Collision (TTC), Comfort (C), and Ego Progress (EP).
- `Extended PDM Score (EPDMS)` — range: percent
  - EPDMS = NC × DAC × DDC × TL × (5×TTC + 2×C + 5×EP + 5×LK + 5×EC) / 22. Adds Traffic Lights Compliance (TL), Driving Direction Compliance (DDC), Lane Keeping Ability (LK), and Extended Comfort (EC) to the base formula.

## Input / output format

**Input**: Multi-view images (front, front-left, front-right) concatenated into 256×1024 resolution, 2 past frames, ego vehicle state (velocity, acceleration), and navigation commands (turning, lane changing, following).

**Output**: 40-waypoint trajectory over 4 seconds (10 Hz sampling), each waypoint defined by (x, y, heading) coordinates.

## Scoring recipe

```python
def compute_pdms(predictions, gold, ego_state):
    # Run NAVSIM benchmark simulator to extract sub-metrics
    # Returns NC, DAC, TTC, C, EP as percentages [0, 100]
    nc, dac, ttc, c, ep = run_pdm_eval(predictions, gold, ego_state)
    # Normalize to [0, 1] for formula multiplication
    nc, dac, ttc, c, ep = nc/100, dac/100, ttc/100, c/100, ep/100
    pdms = nc * dac * (5 * ttc + 2 * c + 5 * ep) / 12.0
    return pdms * 100  # Return as percentage to match reported table values
```

## Common pitfalls

- The sub-metrics (NC, DAC, TTC, etc.) are computed by the NAVSIM benchmark simulator, not analytically from the trajectory alone; users must run the full simulation environment to get accurate values.
- The extended metrics require specific threshold configurations (e.g., τ_D = 0.5m for DDC/LK, τ_A = 0.7m/s² for EC) that must be set exactly as in the benchmark to reproduce scores.
- Comparing against LiDAR-based baselines without noting modality differences is misleading, as this method explicitly relies only on camera inputs and a lightweight ResNet34 backbone.

## Evidence (verbatim from paper)

> The dataset is split into two parts: Navtrain and Navtest, which respectively contain 1192 and 136 scenarios for training/validation and testing. Metrics. For NAVSIM dataset, we evaluate our models based on the PDM score (PDMS) and the Extended PDM Score (EPDMS), which can be formulated as follows: PDMS = NC × DAC × (5×TTC + 2×C + 5×EP)/12, EPDMS = NC × DAC × DDC × TL × (5×TTC + 2×C + 5×EP + 5×LK + 5×EC)/22 where sub-metrics NC, DAC, TTC, C, EP, DDC, TL, LK and EC correspond to the No at-fault Collision, Drivable Area Compliance, Time-to-Collision, Comfort, Ego Progress, Traffic Lights Compliance, Lane Keeping Ability and Extended Comfort.

## Citation

```bibtex
@misc{li2025hydramdpp,
  title={Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2503.12820}
}
```

- arXiv: 2503.12820