Carla Urban Driving Eval

---
name: carla-urban-driving-eval
description: Evaluates an autonomous driving agent's ability to navigate urban environments, avoid dynamic and static obstacles, and handle road blockages under varying weather conditions and unseen towns. Use when the user wants to benchmark on CARLA urban driving benchmark, or asks about evaluating this task. Reports Success Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.11579
  bibtex_key: eraki2022dynamiccil
  confidence: high
---

# carla-urban-driving-eval

> Dynamic Conditional Imitation Learning for Autonomous Driving — Eraqi et al. (2022) (arXiv:2211.11579, 2022)

## What this evaluates

Evaluates an autonomous driving agent's ability to navigate urban environments, avoid dynamic and static obstacles, and handle road blockages under varying weather conditions and unseen towns.

## Datasets

- **CARLA urban driving benchmark** — total 1200; splits: test (1200)

## Metrics

- `Success Rate` **(primary)** — range: percent
  - Percentage of test scenarios where the vehicle reaches the destination within a predetermined deadline (set to the time needed to traverse the shortest route at 10 km/h).
- `Distance to Goal Traveled` — range: percent
  - Average percentage of the total distance to the destination covered by the vehicle across all test scenarios.
- `Average Kilometers Before Infraction` — range: km
  - Mean distance (in km) driven until the first occurrence of a specific infraction (e.g., collision with pedestrian/vehicle/static object, going off-road, invading opposite lane, violating traffic light).

## Input / output format

**Input**: RGB camera images, 32-layer LiDAR point clouds, GPS/waypoint coordinates, traffic light states, and vehicle kinematics from the CARLA simulator.

**Output**: Continuous driving control commands (steering, throttle, brake) and high-level navigational commands (e.g., 'follow lane', 'go left').

## Scoring recipe

```python
def score_eval(scenarios, predictions, gold):
    successes = 0
    dist_to_goal_sum = 0.0
    infraction_dists = {k: [] for k in gold['infraction_types']}
    for i, scenario in enumerate(scenarios):
        traj = predictions[i]
        if reached_destination(traj) and within_deadline(traj, gold['deadline']):
            successes += 1
        dist_to_goal_sum += calc_dist_to_goal_pct(traj, gold['dest'])
        for inf_type in infraction_dists:
            dist = get_first_infraction_dist(traj, inf_type)
            infraction_dists[inf_type].append(dist if dist > 0 else 0.0)
    success_rate = (successes / len(scenarios)) * 100
    avg_dist_to_goal = dist_to_goal_sum / len(scenarios)
    avg_infraction_km = {k: np.mean(v) for k, v in infraction_dists.items()}
    return success_rate, avg_dist_to_goal, avg_infraction_km
```

## Common pitfalls

- CARLA simulator non-determinism (texture loading, pedestrian algorithms) causes result variance across runs.
- The success deadline is set to a low speed (10 km/h) shortest route time, allowing faster models to succeed even if they deviate from the optimal path.
- Results may not exactly match prior publications due to CARLA version updates and rendering changes.

## Evidence (verbatim from paper)

> A test scenario is considered successful if the vehicle reaches the destination within a predetermined deadline. The deadline (maximum allowed time to reach the destination) is set to the time needed to reach the destination along the shortest route at a low speed of 10 km/h as followed in [[14]] and [[10]]. ... The table reports the autonomous driving success rate on different tasks and test conditions, and the average percentage of distance to goal traveled is available between parentheses.

## Citation

```bibtex
@misc{eraki2022dynamiccil,
  title={Dynamic Conditional Imitation Learning for Autonomous Driving},
  author={Eraqi et al. (2022)},
  year={2022},
  note={arXiv:2211.11579}
}
```

- arXiv: 2211.11579