Pseudo Simulation Eval

---
name: pseudo-simulation-eval
description: This evaluation probes the closed-loop planning robustness and causal reasoning of autonomous vehicle controllers by measuring their ability to handle compounding errors and distribution shifts. It combines real-world driving observations with pseudo-synthetic future scenarios generated via neural rendering to approximate interactive simulation without requiring a full physics engine. Use when the user wants to benchmark on nuPlan (navhard subset), or asks about evaluating this task. Reports EPDMS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.04218
  bibtex_key: cao2025pseudosimulation
  confidence: high
---

# pseudo-simulation-eval

> Pseudo-Simulation for Autonomous Driving — Cao et al. (2025) (arXiv:2506.04218, 2025)

## What this evaluates

This evaluation probes the closed-loop planning robustness and causal reasoning of autonomous vehicle controllers by measuring their ability to handle compounding errors and distribution shifts. It combines real-world driving observations with pseudo-synthetic future scenarios generated via neural rendering to approximate interactive simulation without requiring a full physics engine.

## Datasets

- **nuPlan (navhard subset)** — total 5912; splits: Stage 1 (450), Stage 2 (5462); repo https://github.com/autonomousvision/navsim

## Metrics

- `EPDMS` **(primary)** — range: [0, 1]
  - Composite planning score aggregating binary/continuous subscores (NC, DAC, DDC, TLC, EP, TTC, LK, HC, EC) using multiplicative aggregation across two 4-second stages to approximate an 8-second closed-loop score.
- `CLS` — range: [0, 1]
  - Closed-loop score computed by the nuPlan simulator for fully reactive rollouts with privileged perception and HD map access.
- `mIoU` — range: [0, 1]
  - Mean Intersection over Union for Bird’s Eye View segmentation over drivable area, walkway, and vehicle classes.
- `LPIPS` — range: [0, 1]
  - Learned perceptual image patch similarity measuring novel view synthesis fidelity against ground-truth frames.

## Input / output format

**Input**: Per instance, the model receives a sequence of real-world sensor observations (Stage 1) and pseudo-synthetic observations generated via 3D Gaussian Splatting (Stage 2), including BEV images, HD maps, and privileged ground-truth perception data for privileged planners.

**Output**: A planned trajectory or control actions over a 4-second horizon per stage (total 8 seconds), typically at 10Hz frequency.

## Scoring recipe

```python
def compute_epdms(subscores_s1, subscores_s2):
    # subscores: NC, DAC, DDC, TLC, EP, TTC, LK, HC, EC
    # Multiplicative aggregation across stages
    score_s1 = 1.0
    score_s2 = 1.0
    for metric in subscores_s1:
        score_s1 *= metric
    for metric in subscores_s2:
        score_s2 *= metric
    # Hybrid strategy: multiply penalty metrics, average others
    # Paper defaults to multiplicative aggregation for binary-heavy subscores
    epdms = score_s1 * score_s2
    return epdms
```

## Common pitfalls

- Assuming EPDMS is a simple arithmetic mean; the paper explicitly uses multiplicative aggregation to better handle binary penalty metrics.
- Comparing EPDMS directly to nuPlan's CLS without accounting for missing subscores (TLC, LK, EC) in the closed-loop simulator, which biases correlation analysis.
- Treating Stage 2 synthetic observations as independent test cases rather than distribution-shifted variants of Stage 1 real observations.

## Evidence (verbatim from paper)

> We measure the alignment between EPDMS and nuPlan’s closed-loop score (CLS) using Pearson’s linear (r) and Spearman’s rank (ρ) correlation coefficients, as well as the coefficient of determination (R2). Since R2 is calculated by fitting a linear model between EPDMS and CLS, it is equivalent to the square of Pearson’s correlation coefficient here (R2=r2). This assumes that an ideal pseudo-simulation metric should show a linear relationship with closed-loop scores, requiring no adjustments for scale or bias.

## Citation

```bibtex
@misc{cao2025pseudosimulation,
  title={Pseudo-Simulation for Autonomous Driving},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2506.04218}
}
```

- arXiv: 2506.04218