Dsbench Eval

---
name: dsbench-eval
description: Evaluates Vision-Language Models' ability to perceive and reason about safety-critical scenarios in autonomous driving, covering both external environmental hazards (e.g., traffic rules, obstacles, weather) and in-cabin driver states (e.g., fatigue, distraction, emotion). It probes fine-grained hazard recognition, regulatory compliance, and multi-step safety reasoning under diverse, high-risk conditions. Use when the user wants to benchmark on DSBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.14592
  bibtex_key: meng2025dsbench
  confidence: high
---

# dsbench-eval

> Is Your VLM for Autonomous Driving Safety-Ready? A Comprehensive Benchmark for Evaluating External and In-Cabin Risks — Meng et al. (2025) (arXiv:2511.14592, 2025)

## What this evaluates

Evaluates Vision-Language Models' ability to perceive and reason about safety-critical scenarios in autonomous driving, covering both external environmental hazards (e.g., traffic rules, obstacles, weather) and in-cabin driver states (e.g., fatigue, distraction, emotion). It probes fine-grained hazard recognition, regulatory compliance, and multi-step safety reasoning under diverse, high-risk conditions.

## Datasets

- **DSBench** — total 3000; splits: test (3000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered QA pairs. Predictions are compared against ground-truth answers using exact-match or semantic similarity scoring, averaged across all 28 safety subcategories.

## Input / output format

**Input**: Image(s) of driving scenes (external road views and/or in-cabin cockpit views) paired with a natural language question targeting a specific safety dimension or subcategory.

**Output**: Natural language answer (short phrase or sentence) indicating the correct safety action, hazard identification, or driver state classification.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_text(pred) == normalize_text(gold):
        correct += 1
return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models often fail on hypothetical or multi-step reasoning questions that require chaining safety logic, rather than simple object detection.
- In-cabin and external scene confusion: Models trained on siloed datasets may misattribute driver states to external factors or vice versa.
- Pre-annotation bias: The dataset uses GPT-4o for initial categorization, which may propagate specific reasoning patterns or blind spots to evaluated models.

## Evidence (verbatim from paper)

> For evaluation, we curate a subset of 3,000 representative high-risk scenes to form DSBench. In total, we construct 98K QAs covering 28 safety dimensions, combining scale with diversity.

## Citation

```bibtex
@misc{meng2025dsbench,
  title={Is Your VLM for Autonomous Driving Safety-Ready? A Comprehensive Benchmark for Evaluating External and In-Cabin Risks},
  author={Meng et al. (2025)},
  year={2025},
  note={arXiv:2511.14592}
}
```

- arXiv: 2511.14592