Impromptu Vla Diagnostic Eval

---
name: impromptu-vla-diagnostic-eval
description: Diagnoses VLM capabilities in autonomous driving by evaluating perception, prediction, meta-planning via Q&A accuracy, and planning via trajectory prediction L2 error. Use when the user wants to benchmark on Impromptu VLA, or asks about evaluating this task. Reports Q&A Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.23757
  bibtex_key: [REDACTED]
  confidence: high
---

# impromptu-vla-diagnostic-eval

> Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models — Chi et al. (2025) (arXiv:2505.23757, 2025)

## What this evaluates

Diagnoses VLM capabilities in autonomous driving by evaluating perception, prediction, meta-planning via Q&A accuracy, and planning via trajectory prediction L2 error.

## Datasets

- **Impromptu VLA** — total 80000; splits: validation (-1); repo https://github.com/ahydchh/Impromptu-VLA

## Metrics

- `Q&A Accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered questions across four categories: V.R.U. (Vulnerable Road Users), T. Light (Traffic Lights), Dyn. Obj. (Dynamic Objects), and M.P. (Meta-Planning).
- `Traj. Pred. L2 Error (m)` — range: m
  - Euclidean distance in meters between predicted and ground truth trajectories at 1s, 2s, 3s, and 4s horizons, plus average across horizons.

## Input / output format

**Input**: Video clips with planning-oriented Q&A prompts and trajectory ground truth.

**Output**: Text answers to Q&A and predicted trajectory points.

## Scoring recipe

```python
def score_diagnostic(predictions, gold):
    qa_acc = sum(1 for p, g in zip(predictions['qa'], gold['qa']) if p == g) / len(gold['qa'])
    traj_errors = []
    for horizon in [1, 2, 3, 4]:
        pred_pts = predictions['traj'][horizon]
        gt_pts = gold['traj'][horizon]
        l2 = np.sqrt(np.sum((pred_pts - gt_pts)**2, axis=1))
        traj_errors.append(np.mean(l2))
    return qa_acc, traj_errors, np.mean(traj_errors)
```

## Common pitfalls

- Q&A accuracy is broken down into four distinct capability categories rather than a single aggregate score.
- Trajectory prediction horizon extends to 4s in this diagnostic set, differing from the 3s horizon used in the nuScenes open-loop benchmark.

## Evidence (verbatim from paper)

> The quantitative evaluation on the Impromptu VLA validation set, summarized in Table [3], clearly demonstrates that fine-tuning on our dataset can transforms to all crucial aspects of autonomous driving, including perception, prediction, reasoning for planning, and the planned trajectory. Accuracy ↑ is reported for perception (V.R.U., T. Light), prediction (Dyn. Obj.), meta-planning (M.P.) and Planning (L2).

## Citation

```bibtex
@misc{chi2025impromptu,
  title={Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models},
  author={Chi et al. (2025)},
  year={2025},
  note={arXiv:2505.23757}
}
```

- arXiv: 2505.23757