Surds Eval

---
name: surds-eval
description: This benchmark evaluates fine-grained spatial understanding and reasoning capabilities of vision-language models in real-world driving scenarios. It probes six distinct spatial dimensions: orientation (Yaw), pixel-level localization, depth estimation, pairwise distance, lateral ordering, and front-back relations. Use when the user wants to benchmark on SURDS, or asks about evaluating this task. Reports Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.13112
  bibtex_key: guo2024surds
  confidence: high
---

# surds-eval

> SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models — Guo et al. (2024) (arXiv:2411.13112, 2024)

## What this evaluates

This benchmark evaluates fine-grained spatial understanding and reasoning capabilities of vision-language models in real-world driving scenarios. It probes six distinct spatial dimensions: orientation (Yaw), pixel-level localization, depth estimation, pairwise distance, lateral ordering, and front-back relations.

## Datasets

- **SURDS** — total ?; splits: test (-1); repo https://github.com/XiandaGuo/Drive-MLLM

## Metrics

- `Score` **(primary)** — range: [0, 100]
  - For the Pixel Localization task, a centerness-based metric is used. For the other five tasks (Yaw, Depth, Dis, L/R, F/B), a prediction receives 1 if it exactly matches the ground-truth answer and 0 otherwise. Each task's score is the average over all N QA pairs, scaled to a percentage. The final Score is the unweighted average of the six individual task scores.

## Input / output format

**Input**: A driving scene image paired with a standardized instruction/question and strict output format constraints.

**Output**: A structured response that must strictly adhere to a predefined format specified by the benchmark.

## Scoring recipe

```python
def compute_score(predictions, golds):
    task_scores = {}
    for task in ['Yaw', 'Pixel', 'Depth', 'Dis', 'L/R', 'F/B']:
        if task == 'Pixel':
            task_scores[task] = compute_centerness_metric(predictions[task], golds[task])
        else:
            matches = [1 if p == g else 0 for p, g in zip(predictions[task], golds[task])]
            task_scores[task] = (sum(matches) / len(matches)) * 100
    return sum(task_scores.values()) / len(task_scores)
```

## Common pitfalls

- The Pixel Localization task uses a centerness-based metric rather than exact-match accuracy, unlike the other five tasks.
- The overall Score is a simple unweighted average across six spatial dimensions, which can mask significant performance gaps between single-object and multi-object reasoning tasks.
- Models must strictly follow the predefined output format; failure to do so prevents parsing and evaluation.

## Evidence (verbatim from paper)

> The Score column represents the average performance across these six metrics. For the Pixel Localization Estimation task, we adopt a centerness-based metric. For other tasks, a prediction receives a score of 1 if it matches the ground-truth answer, and 0 otherwise. Given N QA pairs, the metric score for each task is computed as the average over all N pairs. The final overall score is the average of all individual task scores.

## Citation

```bibtex
@misc{guo2024surds,
  title={SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models},
  author={Guo et al. (2024)},
  year={2024},
  note={arXiv:2411.13112}
}
```

- arXiv: 2411.13112