Nplanqa Eval

---
name: nplanqa-eval
description: This benchmark evaluates multi-modal large language models on their ability to understand complex driving scenes using multi-view and multi-frame inputs. It probes three core capabilities: road environment perception, spatial relations recognition, and ego-centric reasoning through visual question answering. Use when the user wants to benchmark on NuPlanQA-Eval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.12772
  bibtex_key: park2025nplanqa
  confidence: high
---

# nplanqa-eval

> NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models — Park et al. (2025) (arXiv:2503.12772, 2025)

## What this evaluates

This benchmark evaluates multi-modal large language models on their ability to understand complex driving scenes using multi-view and multi-frame inputs. It probes three core capabilities: road environment perception, spatial relations recognition, and ego-centric reasoning through visual question answering.

## Datasets

- **NuPlanQA-Eval** — total ?; splits: test (-1); repo https://github.com/sungyeonparkk/NuPlanQA

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the number of correct responses divided by the total number of questions, multiplied by 100 to yield a percentage. The benchmark uses multiple-choice questions with four answer options, establishing a 25% random guessing baseline.

## Input / output format

**Input**: Structured multi-view images (arranged into a single composite image with camera view markers) or single front-view images, optionally combined with historical frames (up to 3 past frames for ~1.5s of video). Accompanied by a multiple-choice question with four options.

**Output**: A single selected answer choice from the four provided options.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models frequently fail to prioritize the correct traffic light when multiple signals are present in the scene.
- Spatial relations recognition is significantly harder than other tasks because it relies purely on visual cues and temporal reasoning across frames, lacking the aid of control signals or static pretraining biases.
- Performance drops substantially when models are restricted to single-frame inputs, as temporal context is critical for ego-centric reasoning and spatial tracking.

## Evidence (verbatim from paper)

> Since our benchmark consists of multiple-choice questions with four answer options, we adopt accuracy as the evaluation metric, where random guessing yields a baseline accuracy of 25%.

## Citation

```bibtex
@misc{park2025nplanqa,
  title={NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2503.12772}
}
```

- arXiv: 2503.12772