Carla No Crash Eval

---
name: carla-no-crash-eval
description: Evaluates autonomous driving agents' robustness and generalization in a simulated urban environment, specifically testing performance on familiar and unseen town layouts. Use when the user wants to benchmark on CARLA NoCrash benchmark, or asks about evaluating this task. Reports Driving Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.10458
  bibtex_key: qu2026foundrl
  confidence: high
---

# carla-no-crash-eval

> Found-RL: foundation model-enhanced reinforcement learning for autonomous driving — Qu et al. (2026) (arXiv:2602.10458, 2026)

## What this evaluates

Evaluates autonomous driving agents' robustness and generalization in a simulated urban environment, specifically testing performance on familiar and unseen town layouts.

## Datasets

- **CARLA NoCrash benchmark** — total ?; splits: train (-1), test (-1)

## Metrics

- `Driving Score` **(primary)** — range: [0, 1]
  - Product of route completion percentage and an infraction penalty discount factor.
- `Success Rate` — range: [0, 1]
  - Fraction of episodes successfully completed without triggering terminal failure conditions.
- `Route Completion` — range: [0, 100] percent
  - Percentage of the reference route distance successfully traversed.
- `Infraction Penalty` — range: [0, 1]
  - Discount factor aggregating penalties for traffic rule violations (e.g., red lights, collisions).
- `Collisions of Vehicle` — range: other
  - Number of vehicle collisions normalized by completed route length.
- `Collisions of Pedestrian` — range: other
  - Number of pedestrian collisions normalized by completed route length.
- `Red Light` — range: other
  - Number of red light violations over the completed route length.
- `Return` — range: other
  - Accumulated rewards over the entire episode.
- `Icell` — range: other
  - Estimated battery cell current based on a single-cell equivalent circuit model.
- `Fuel Rate` — range: other
  - Instantaneous fuel consumption derived from a longitudinal road-load model.
- `Speed` — range: other
  - Average driving speed over the episode.

## Input / output format

**Input**: BEV image (192x192x3) and text prompt for VLMs; BEV masks (96x96x15) and compact state vector (vehicle motion, last-step controls, traffic context) for RL agents.

**Output**: Continuous 3D action (throttle, steer, brake) for VLMs; continuous 2D action (signed longitudinal, steer) for RL agents.

## Scoring recipe

```python
Driving Score = Route Completion * Infraction Penalty.
Episode terminates on collision, route deviation, traffic violation, or completion.
Metrics are averaged over 3 independent runs with different random seeds.
```

## Common pitfalls

- Evaluation uses a stricter termination logic than the original ROACH/Leaderboard protocol, including tighter route adherence checks and additional failure triggers.
- Physics for BEV-invisible static objects are disabled to prevent unobservable collision terminations that disrupt credit assignment.

## Evidence (verbatim from paper)

> To comprehensively evaluate the performance of the driving agents, we use a set of metrics organized into four primary categories: Comprehensive, Route, Energy, and Safety. The Comprehensive metrics include Return, which represents the accumulated rewards over the entire episode; Driving Score, defined as the product of route completion and the infraction penalty; and Infraction Penalty, a discount factor aggregating all triggered infractions.

## Citation

```bibtex
@misc{qu2026foundrl,
  title={Found-RL: foundation model-enhanced reinforcement learning for autonomous driving},
  author={Qu et al. (2026)},
  year={2026},
  note={arXiv:2602.10458}
}
```

- arXiv: 2602.10458