Drivingvqa Eval

---
name: drivingvqa-eval
description: Evaluates a vision-language model's ability to perform multi-label multiple-choice question answering on real-world driving scenarios, requiring precise visual grounding and spatial reasoning to select all correct answers from a set of options. Use when the user wants to benchmark on DrivingVQA, or asks about evaluating this task. Reports exam score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.04671
  bibtex_key: corbiere2025rivcot
  confidence: high
---

# drivingvqa-eval

> Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios — Corbière et al. (2025) (arXiv:2501.04671, 2025)

## What this evaluates

Evaluates a vision-language model's ability to perform multi-label multiple-choice question answering on real-world driving scenarios, requiring precise visual grounding and spatial reasoning to select all correct answers from a set of options.

## Datasets

- **DrivingVQA** — total ?; splits: train (-1), test (-1)

## Metrics

- `exam score` **(primary)** — range: percent
  - Equivalent to exact-match accuracy for multi-label classification. A question scores 1 only if all correct answer options are selected, and 0 otherwise. The final score is the percentage of correctly answered questions.
- `F1-Score` — range: percent
  - Multi-label classification F1-score computed over all answer options across the test set.

## Input / output format

**Input**: Image, question text, and list of possible answers. Optionally augmented with entity labels, bounding box coordinates, and/or visual crops of relevant entities.

**Output**: Predicted answer(s), optionally preceded by a chain-of-thought explanation. For RIV-CoT, the output interleaves entity bounding boxes, visual patches, and reasoning steps before the final answer.

## Scoring recipe

```python
def compute_exam_score(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if set(p) == set(g))
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- The exam score is an exact-match metric for multi-label answers, not standard single-label accuracy. Missing or adding any option results in a 0 for that question.
- Results are averaged over 5 random seeds and reported with standard deviation; single-run evaluations will not match paper tables.
- Visual inputs are strictly required; evaluating without images causes a ~26 point drop, so text-only baselines are invalid for this benchmark.

## Evidence (verbatim from paper)

> Evaluation metrics. The performance on the DrivingVQA test split is measured with the exam score, analogous to the real driving theory score used to evaluate candidates: all correct answers must be selected to score a question correctly. Note that this score is equivalent to the exact match ratio. Since the task is analogous to multi-label classification, we also report the F1-Score.

## Citation

```bibtex
@misc{corbiere2025rivcot,
  title={Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios},
  author={Corbière et al. (2025)},
  year={2025},
  note={arXiv:2501.04671}
}
```

- arXiv: 2501.04671