Carla Autonomous Driving Eval

---
name: carla-autonomous-driving-eval
description: Evaluates the end-to-end inference latency of an autonomous driving pipeline that runs parallel reinforcement learning and object detection models in a simulated urban environment. It measures how efficiently the middleware handles communication and computation overhead during real-time sensor processing and action fusion. Use when the user wants to benchmark on CARLA simulator, or asks about evaluating this task. Reports inference_latency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.01799
  bibtex_key: kwok2024hprm
  confidence: high
---

# carla-autonomous-driving-eval

> HPRM: High-Performance Robotic Middleware for Intelligent Autonomous Systems — Kwok et al. (2024) (arXiv:2412.01799, 2024)

## What this evaluates

Evaluates the end-to-end inference latency of an autonomous driving pipeline that runs parallel reinforcement learning and object detection models in a simulated urban environment. It measures how efficiently the middleware handles communication and computation overhead during real-time sensor processing and action fusion.

## Datasets

- **CARLA simulator** — total ?; splits: test (400)

## Metrics

- `inference_latency` **(primary)** — range: ms
  - Sum of communication time and inference time per frame, explicitly excluding physics computation time.

## Input / output format

**Input**: BEV and RGB camera images from the CARLA simulator, fed into parallel PPO policy and YOLO object detection models.

**Output**: Final control action determined by a Fusion reactor/node after synchronizing and combining outputs from the PPO and YOLO models.

## Scoring recipe

```python
latencies = []
for step in range(100):  # warm-up
    run_pipeline()
for step in range(400):  # benchmark
    t_start = time.now()
    action = run_pipeline()
    t_end = time.now()
    latencies.append(t_end - t_start - physics_time)
return mean(latencies)
```

## Common pitfalls

- Including CARLA physics computation time in the latency measurement, which inflates results and misrepresents middleware performance.
- Failing to discard the initial 100 warm-up steps, leading to skewed latency due to cold-start effects and JIT/model loading overhead.
- Comparing centralized and decentralized coordination without accounting for synchronization overhead differences that diminish as object size grows.

## Evidence (verbatim from paper)

> Inference latency is measured as the sum of communication time and inference time. We found that running PPO policy inference in CPU and YOLO in GPU led to a slight performance increase due to full utilization of compute resources, and was implemented across the benchmark. The box plot in Figure [6] illustrates the inference latency measured when running the CARLA benchmark across 400 environment step frames after 100 warm-up steps with HPRM and ROS2 Humble. To obtain a more accurate measure of inference latency, our benchmarks exclude the time CARLA spends computing physics.

## Citation

```bibtex
@misc{kwok2024hprm,
  title={HPRM: High-Performance Robotic Middleware for Intelligent Autonomous Systems},
  author={Kwok et al. (2024)},
  year={2024},
  note={arXiv:2412.01799}
}
```

- arXiv: 2412.01799