Navsim Epdms Eval

---
name: navsim-epdms-eval
description: Evaluates closed-loop end-to-end autonomous driving performance in safety-critical and diverse real-world scenarios. It measures collision avoidance, rule compliance, progress, and comfort under reactive traffic conditions. Use when the user wants to benchmark on navhard, navtest, or asks about evaluating this task. Reports EPDMS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.23369
  bibtex_key: tian2025simscale
  confidence: high
---

# navsim-epdms-eval

> SimScale: Learning to Drive via Real-World Simulation at Scale — Haochen Tian et al. (2025) (arXiv:2511.23369, 2025)

## What this evaluates

Evaluates closed-loop end-to-end autonomous driving performance in safety-critical and diverse real-world scenarios. It measures collision avoidance, rule compliance, progress, and comfort under reactive traffic conditions.

## Datasets

- **navhard** — total 4408; splits: stage1 (244), stage2 (4164)
- **navtest** — total 12146; splits: test (12146)

## Metrics

- `EPDMS` **(primary)** — range: percent
  - EPDMS = (∏ S_m for m in M_pen) * (∑ w_m S_m / ∑ w_m for m in M_avg). Penalty terms (M_pen) include No-at-fault Collisions (NC), Drivable Area Compliance (DAC), Driving Direction Compliance (DDC), and Traffic Light Compliance (TLC). Weighted average terms (M_avg) include Time-to-Collision (TTC), Ego Progress (EP), Lane Keeping (LK), History Comfort (HC), and extended comfort (EC). Weights w_m are defined by the NAVSIM benchmark.

## Input / output format

**Input**: Multi-view camera images (unified to 2048×512 resolution) without LiDAR inputs. The model processes these to generate closed-loop driving trajectories.

**Output**: Closed-loop driving actions/trajectories evaluated over a simulation rollout.

## Scoring recipe

```python
def compute_epdms(sub_metrics):
    pen_terms = ['NC', 'DAC', 'DDC', 'TLC']
    avg_terms = ['TTC', 'EP', 'LK', 'HC', 'EC']
    weights = {m: 1.0 for m in avg_terms}  # weights per NAVSIM spec
    penalty_score = 1.0
    for m in pen_terms:
        penalty_score *= sub_metrics[m]
    avg_score = sum(weights[m] * sub_metrics[m] for m in avg_terms) / sum(weights.values())
    return penalty_score * avg_score
```

## Common pitfalls

- navhard uses a two-stage aggregation where Stage 2 scenarios are synthetically generated from Stage 1 failures.
- Penalties are excluded if the human expert driver also fails in that scenario.
- Regression-based planners may degrade under high simulation-to-real ratios due to multi-modal supervision, unlike diffusion models.

## Evidence (verbatim from paper)

> The two benchmarks share a rule-based planning metric, EPDMS [44], with several sub-metrics: EPDMS = (∏ S_m for m in M_pen) * (∑ w_m S_m / ∑ w_m for m in M_avg), where S_m is the sub-metric: penalty terms set M_pen includes No-at-fault Collisions (NC), Drivable Area Compliance (DAC), Driving Direction Compliance (DDC), and Traffic Light Compliance (TLC); weighted average terms set M_avg includes Time-to-Collision (TTC), Ego Progress (EP), Lane Keeping (LK), History Comfort (HC), and extended comfort (EC).

## Citation

```bibtex
@misc{tian2025simscale,
  title={SimScale: Learning to Drive via Real-World Simulation at Scale},
  author={Haochen Tian et al. (2025)},
  year={2025},
  note={arXiv:2511.23369}
}
```

- arXiv: 2511.23369