Lingoqa Eval

---
name: lingoqa-eval
description: Evaluates vision-language models on autonomous driving video question answering, testing their ability to understand temporal visual context, describe scenes, predict actions, and justify answers based on driving scenarios. Use when the user wants to benchmark on LingoQA, or asks about evaluating this task. Reports Ling-Judge.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.07092
  bibtex_key: cai2026efficient
  confidence: high
---

# lingoqa-eval

> Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression — Cai et al. (2026) (arXiv:2601.07092, 2026)

## What this evaluates

Evaluates vision-language models on autonomous driving video question answering, testing their ability to understand temporal visual context, describe scenes, predict actions, and justify answers based on driving scenarios.

## Datasets

- **LingoQA** — total 419900; splits: train (-1), val (-1), test (-1)

## Metrics

- `Ling-Judge` **(primary)** — range: [0, 100]
  - A semantic similarity score computed by a fine-tuned BERT-based classifier that compares the generated answer against the ground truth answer. Scores are reported on a 0-100 scale.
- `BLEU` — range: [0, 1]
  - Standard N-gram based metric measuring the precision of n-gram overlaps between the generated answer and ground truth, typically averaged across n=1 to 4.
- `FLOPs (%)` — range: percent
  - Relative computational cost measured as a percentage of the baseline full-frame (5-frame) inference cost.

## Input / output format

**Input**: A sequence of 1 to 5 video frames from an autonomous driving scenario, paired with a natural language question.

**Output**: A natural language text answer generated by the model.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    bleu_scores = [sentence_bleu(g, p) for g, p in zip(golds, predictions)]
    ling_scores = []
    for g, p in zip(golds, predictions):
        score = bert_classifier.predict_semantic_similarity(g, p)
        ling_scores.append(score)
    return {'BLEU': mean(bleu_scores), 'Ling-Judge': mean(ling_scores)}
```

## Common pitfalls

- Ling-Judge requires a specific fine-tuned BERT classifier; using a vanilla BERT or standard semantic similarity models will yield incorrect scores.
- BLEU scores are inherently low (10-15) for open-ended video QA; comparing absolute BLEU values across different datasets or tasks is misleading.
- Performance is highly sensitive to the number of input frames (1 vs 5); results must be compared within the same frame-count setting.

## Evidence (verbatim from paper)

> Besides the common N-Gram-based metrics such as BLEU [[11]], LingoQA introduces a semantic-based metric, $Lingo$-$Judge$, such that it adopt fine-tuned Bert-based model as classifier to evaluate the semantic similarity between the generated answer and the ground truth answer.

## Citation

```bibtex
@misc{cai2026efficient,
  title={Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression},
  author={Cai et al. (2026)},
  year={2026},
  note={arXiv:2601.07092}
}
```

- arXiv: 2601.07092