Recogdrive Eval

---
name: recogdrive-eval
description: Evaluates an end-to-end autonomous driving agent's ability to generate safe, comfortable, and efficient driving trajectories using only camera inputs. It probes the model's closed-loop planning capabilities, safety-critical scenario handling, and visual reasoning in complex urban environments. Use when the user wants to benchmark on NAVSIM, Bench2Drive, or asks about evaluating this task. Reports PDMS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.08052
  bibtex_key: li2025recogdrive
  confidence: high
---

# recogdrive-eval

> ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving — Yongkang Li et al. (arXiv:2506.08052, 2025)

## What this evaluates

Evaluates an end-to-end autonomous driving agent's ability to generate safe, comfortable, and efficient driving trajectories using only camera inputs. It probes the model's closed-loop planning capabilities, safety-critical scenario handling, and visual reasoning in complex urban environments.

## Datasets

- **NAVSIM** — total 1328; splits: train (1192), test (136)
- **Bench2Drive** — total 220; splits: test (220)

## Metrics

- `PDMS` **(primary)** — range: [0, 100]
  - Planning Driving Metric Score, a composite closed-loop metric from the nuPlan/NAVSIM protocol that aggregates safety, comfort, efficiency, and progress scores into a single normalized value.
- `NC` — range: percent
  - No-Collision rate, measuring the percentage of simulation steps without collisions.
- `DAC` — range: percent
  - Drivable Area Compliance, measuring the percentage of time the vehicle stays within drivable boundaries.
- `TTC` — range: percent
  - Time-to-Collision metric, evaluating proximity to potential collisions over the trajectory.
- `Comf` — range: percent
  - Comfort score, penalizing harsh acceleration, braking, and steering maneuvers.
- `EP` — range: percent
  - Efficiency Progress, measuring how much of the route is completed relative to a baseline.
- `DS` — range: percent
  - Driving Score on Bench2Drive, a CARLA leaderboard metric combining efficiency, comfort, and success rate.
- `DriveBench Avg` — range: percent
  - Average score across perception, prediction, planning, and behavior sub-tasks on the DriveLM/DriveBench VQA benchmarks.

## Input / output format

**Input**: Camera images processed via dynamic resolution preprocessing, along with standard driving environment observations (ego state, map, traffic participants) implicit in the NAVSIM/Bench2Drive protocols.

**Output**: Continuous, feasible driving trajectories generated by a diffusion planner.

## Scoring recipe

```python
def evaluate_closed_loop(model, dataset, sim_env):
    scores = []
    for scene in dataset:
        traj = model.predict(scene.images, scene.ego_state)
        sim_result = sim_env.run(traj)
        nc = sim_result.no_collision_rate
        dac = sim_result.drivable_area_compliance
        ttc = sim_result.time_to_collision
        comf = sim_result.comfort_score
        ep = sim_result.efficiency_progress
        pdms = composite_metric(nc, dac, ttc, comf, ep)
        scores.append(pdms)
    return mean(scores)
```

## Common pitfalls

- The paper explicitly uses only camera input, unlike many baselines that use LiDAR; failing to match input modality leads to unfair comparisons.
- Evaluation is strictly closed-loop (simulated driving), not open-loop trajectory matching; metrics like PDMS depend on the agent's continuous interaction with the environment.
- Bench2Drive uses CARLA's leaderboard protocol with specific safety-critical scenarios; results are not directly comparable to open-loop or different closed-loop benchmarks.

## Evidence (verbatim from paper)

> We evaluate primarily on two challenging benchmarks: NAVSIM(Dauner et al., [2025]) and Bench2Drive(Jia et al., [2024]). NAVSIM is a planning-oriented autonomous driving dataset built on OpenScene*(Contributors, [2023])*, a redistribution of nuPlan*(Caesar et al., [2021])*. The dataset is split into navtrain (1,192 training scenes) and navtest (136 evaluation scenes). Bench2Drive is a CARLA-based benchmark composed of 220 short routes, each containing a distinct, safety-critical scenario. ReCogDrive achieves a PDMS of 90.8, establishing a new state-of-the-art.

## Citation

```bibtex
@misc{li2025recogdrive,
  title={ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving},
  author={Yongkang Li et al.},
  year={2025},
  note={arXiv:2506.08052}
}
```

- arXiv: 2506.08052