Racecar Eval

---
name: racecar-eval
description: Evaluates high-speed autonomous driving capabilities including precise localization, long-range object detection and tracking, and robust mapping/SLAM under extreme dynamic conditions (up to 170 mph). The protocol benchmarks how well models maintain accuracy and latency when processing multi-modal sensor data at racing speeds where motion blur, sensor dropout, and rapid ego-motion are prevalent. Use when the user wants to benchmark on RACECAR, or asks about evaluating this task. Reports Average Precision (AP).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.03252
  bibtex_key: kulkarni2023racecar
  confidence: high
---

# racecar-eval

> RACECAR -- The Dataset for High-Speed Autonomous Racing — Kulkarni et al. (2023) (arXiv:2306.03252, 2023)

## What this evaluates

Evaluates high-speed autonomous driving capabilities including precise localization, long-range object detection and tracking, and robust mapping/SLAM under extreme dynamic conditions (up to 170 mph). The protocol benchmarks how well models maintain accuracy and latency when processing multi-modal sensor data at racing speeds where motion blur, sensor dropout, and rapid ego-motion are prevalent.

## Datasets

- **RACECAR** — total ?; splits: train (-1), test (-1); repo https://github.com/linklab-uva/RACECAR_DATA

## Metrics

- `Average Precision (AP)` **(primary)** — range: percent
  - Standard object detection metric computing the area under the precision-recall curve. The paper reports AP averaged over classes, evaluated separately across four distance ranges (0-20m, 20-40m, 40-60m, 60m-Inf) to capture high-speed range degradation.
- `Localization Error` — range: meters
  - Measured as the distance the vehicle travels between GNSS updates or the deviation of the fused pose estimate from ground truth. Reported in meters to quantify blind-spot distance and tracking stability at 100+ mph.

## Input / output format

**Input**: Synchronized multi-modal sensor streams including 3D LiDAR point clouds, radar detections (angle, distance, velocity), camera images (6 global shutter cameras), GNSS coordinates, and IMU data (accelerometer/gyroscope at 125 Hz, wheel rotation at 100 Hz).

**Output**: For detection: 3D bounding boxes or BEV detections with class labels, confidence scores, and distance estimates. For localization: vehicle pose [x, y, z, θ] in world frame at 100 Hz. For mapping: 3D point cloud map or pose trajectory with motion distortion compensation.

## Scoring recipe

```python
def compute_ap(predictions, ground_truth, iou_thresh=0.5):
    aps = {}
    ranges = [(0, 20), (20, 40), (40, 60), (60, float('inf'))]
    for low, high in ranges:
        pred_f = [p for p in predictions if low <= p['distance'] < high]
        gt_f = [g for g in ground_truth if low <= g['distance'] < high]
        precisions, recalls = compute_pr_curve(pred_f, gt_f, iou_thresh)
        ap = interpolate_ap(precisions, recalls)
        aps[f'{low}-{high}m'] = ap
    return {'overall_ap': sum(aps.values())/len(aps), 'range_aps': aps}
```

## Common pitfalls

- High-speed motion blur and LiDAR dropout at 100+ mph significantly degrade detection and mapping performance; models must compensate for scan distortion before matching.
- GNSS-only localization updates at 20 Hz are insufficient for racing; EKF fusion at 100 Hz is required to reduce blind-spot distance from ~2m to ~0.4m.
- Distance-dependent performance drops sharply beyond 60m; reporting overall AP without range breakdown hides critical high-speed safety limitations.

## Evidence (verbatim from paper)

> Inference on point clouds using PointPillars resulted in maximum detections at distances up to 110 m. However, the average precision of detections dropped approximately 40% for distances over 60 m. A benchmark challenge is to improve the detection range and reliability.

## Citation

```bibtex
@misc{kulkarni2023racecar,
  title={RACECAR -- The Dataset for High-Speed Autonomous Racing},
  author={Kulkarni et al. (2023)},
  year={2023},
  note={arXiv:2306.03252}
}
```

- arXiv: 2306.03252