Roadscapesqa Eval

---
name: roadscapesqa-eval
description: Evaluates vision-language models on visual question answering for Indian road scenes. It probes capabilities in object counting, object description, and surrounding scene description across diverse driving environments. Use when the user wants to benchmark on RoadscapesQA, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.12877
  bibtex_key: iyer2026roadscapesqa
  confidence: high
---

# roadscapesqa-eval

> RoadscapesQA: A Multitask, Multimodal Dataset for Visual Question Answering on Indian Roads — Iyer et al. (2026) (arXiv:2602.12877, 2026)

## What this evaluates

Evaluates vision-language models on visual question answering for Indian road scenes. It probes capabilities in object counting, object description, and surrounding scene description across diverse driving environments.

## Datasets

- **RoadscapesQA** — total 9000; splits: test (1500); repo https://github.com/vijpandaturtle/roadscapes

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - Proportion of instances where the model's predicted answer exactly matches the ground truth answer character-for-character.
- `cosine similarity` — range: other
  - Cosine similarity between sentence embeddings of the model's free-form output and the ground truth answer, computed using the all-MiniLM-L6-v2 embedding model.

## Input / output format

**Input**: Image-question pairs (a single image and a text question)

**Output**: Free-form text response

## Scoring recipe

```python
def compute_metrics(predictions, golds, tasks):
    scores = {}
    for task in ['Object Counting', 'Object Description', 'Surrounding Description']:
        preds = [p for p, t in zip(predictions, tasks) if t == task]
        golds_t = [g for g, t in zip(golds, tasks) if t == task]
        if task == 'Object Counting':
            correct = sum(1 for p, g in zip(preds, golds_t) if p.strip() == str(g))
            scores[task] = correct / len(preds)
        else:
            from sentence_transformers import SentenceTransformer
            import numpy as np
            model = SentenceTransformer('all-MiniLM-L6-v2')
            p_emb = model.encode(preds)
            g_emb = model.encode(golds_t)
            sims = np.sum(p_emb * g_emb, axis=1) / (np.linalg.norm(p_emb, axis=1) * np.linalg.norm(g_emb, axis=1))
            scores[task] = np.mean(sims)
    return scores
```

## Common pitfalls

- Models are evaluated strictly in a zero-shot setting without fine-tuning, which may not reflect their full capability or optimal performance on this domain.
- Exact-match accuracy for object counting is highly sensitive to formatting differences (e.g., 'two' vs '2' or extra whitespace), potentially underestimating semantic correctness.
- Cosine similarity relies on a fixed sentence embedding model (all-MiniLM-L6-v2) that may not capture complex spatial or contextual nuances in driving scenes as effectively as human evaluation or LLM-as-a-judge approaches.

## Evidence (verbatim from paper)

> For the Object Counting task, we employ exact-match accuracy as the primary evaluation metric, following established practice*Chen et al. ([2024])*. The Object Description and Surrounding Description tasks are evaluated using cosine similarity between sentence embeddings, specifically utilizing the all-MiniLM-L6-v2 model*Wang and others ([2020])*.

## Citation

```bibtex
@misc{iyer2026roadscapesqa,
  title={RoadscapesQA: A Multitask, Multimodal Dataset for Visual Question Answering on Indian Roads},
  author={Iyer et al. (2026)},
  year={2026},
  note={arXiv:2602.12877}
}
```

- arXiv: 2602.12877