Carla Real Traffic Scenarios Eval

---
name: carla-real-traffic-scenarios-eval
description: Evaluates the ability of reinforcement learning policies to execute tactical driving maneuvers (e.g., lane changes, roundabout navigation) in a simulated environment mapped from real-world traffic data. It probes how observation modalities and reward structures impact policy generalization and success rates across diverse driving scenarios. Use when the user wants to benchmark on NGSIM, openDD, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.11329
  bibtex_key: osinski2020carla
  confidence: high
---

# carla-real-traffic-scenarios-eval

> CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving — Osiński et al. (2020) (arXiv:2012.11329, 2020)

## What this evaluates

Evaluates the ability of reinforcement learning policies to execute tactical driving maneuvers (e.g., lane changes, roundabout navigation) in a simulated environment mapped from real-world traffic data. It probes how observation modalities and reward structures impact policy generalization and success rates across diverse driving scenarios.

## Datasets

- **NGSIM** — total ?; splits: test (-1); repo https://github.com/deepsense-ai/carla-real-traffic-scenarios
- **openDD** — total ?; splits: test (-1); repo https://github.com/deepsense-ai/carla-real-traffic-scenarios

## Metrics

- `success rate` **(primary)** — range: [0, 1]
  - The proportion of test scenarios where the policy successfully completes the target maneuver without crashing or violating constraints.

## Input / output format

**Input**: Per instance: a tactical driving scenario (lane change or roundabout) with observations provided as either a bird’s-eye view tensor (186×150×5), visual camera feed, or LiDAR & camera data.

**Output**: Per instance: a sequence of continuous driving actions (steering, acceleration, braking) generated by the RL policy to navigate the scenario.

## Scoring recipe

```python
def compute_success_rate(predictions, gold):
    successful = 0
    total = len(predictions)
    for traj in predictions:
        if traj.completed_maneuver and not traj.crashed:
            successful += 1
    return successful / total
```

## Common pitfalls

- Confusing training success rate with test success rate, as the paper notes dense rewards converge faster but sparse rewards may generalize differently.
- Assuming LiDAR & camera modalities are sufficient for roundabouts, as the paper shows they lack road curvature data and perform worse on openDD.
- Overlooking that success is defined by maneuver completion without crashes, not just proximity to a target.

## Evidence (verbatim from paper)

> Table III: Success rates on test scenarios for CRTS scenarios with bird’s-eye view input. Again we can compare the success rate from training (Figure 4) and test (Table III). The dense and no failure penalty look very similar during training, sparse reward takes more time to converge, but seems to reach the same performance. In the evaluation however, dense reward is the clear winner.

## Citation

```bibtex
@misc{osinski2020carla,
  title={CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving},
  author={Osiński et al. (2020)},
  year={2020},
  note={arXiv:2012.11329}
}
```

- arXiv: 2012.11329