Neuroncap Closed Loop Eval

---
name: neuroncap-closed-loop-eval
description: Evaluates closed-loop autonomous driving performance in simulated real-world scenarios, measuring safety and planning efficiency through collision avoidance and impact speed mitigation. Use when the user wants to benchmark on nuScenes, or asks about evaluating this task. Reports NeuroNCAP Score (NNS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.23757
  bibtex_key: chi2025impromptu
  confidence: high
---

# neuroncap-closed-loop-eval

> Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models — Chi et al. (2025) (arXiv:2505.23757, 2025)

## What this evaluates

Evaluates closed-loop autonomous driving performance in simulated real-world scenarios, measuring safety and planning efficiency through collision avoidance and impact speed mitigation.

## Datasets

- **nuScenes** — total ?; splits: test (-1)

## Metrics

- `NeuroNCAP Score (NNS)` **(primary)** — range: [0, 5]
  - 5.0 if no collision occurs; otherwise 4.0 * max(0, 1 - v_i / v_r), where v_i is actual impact speed and v_r is reference impact speed without evasive action. Capped at 4.0 for collisions.
- `Collision rate (%)` — range: percent
  - Percentage of scenarios resulting in a collision, categorized by interaction type (Avg, Stat, Frontal, Side).

## Input / output format

**Input**: Video frames and sensor data from nuScenes driving scenarios.

**Output**: Vehicle control actions (steering, acceleration, braking) over time.

## Scoring recipe

```python
def score_neuroncap(predictions, gold):
    collisions = [p['collides'] for p in predictions]
    collision_rate = sum(collisions) / len(collisions) * 100
    scores = []
    for p in predictions:
        if not p['collides']:
            scores.append(5.0)
        else:
            vi = p['impact_speed']
            vr = p['reference_speed']
            scores.append(4.0 * max(0, 1 - vi / vr))
    return sum(scores) / len(scores), collision_rate
```

## Common pitfalls

- NNS maxes at 5.0 only for zero collisions; collision scenarios are capped at 4.0.
- Metrics are reported across four interaction categories (Avg, Stat, Frontal, Side), not just a single global average.

## Evidence (verbatim from paper)

> The NNS is computed, in the spirit of a 5-star rating system, as follows: a score of 5.0 is achieved if no collision occurs; otherwise, the score is 4.0⋅max(0,1−vi/vr), where vi is the actual impact speed (magnitude of relative velocity between the ego-vehicle and the colliding actor) and vr is the reference impact speed that would occur if no evasive action were performed. Collision rates, on the other hand, directly track the percentage of scenarios resulting in a collision.

## Citation

```bibtex
@misc{chi2025impromptu,
  title={Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models},
  author={Chi et al. (2025)},
  year={2025},
  note={arXiv:2505.23757}
}
```

- arXiv: 2505.23757