Vqa Benchmarks Eval

---
name: vqa-benchmarks-eval
description: Evaluates vision-language models' ability to answer questions across general knowledge, OCR, mathematics, and science domains using few-shot in-context learning. It also probes the model's capacity to attend to interleaved image-text contexts through a 'cheat test' protocol. Use when the user wants to benchmark on TextVQA, OKVQA, MathVista, MathVision, MathVerse, ScienceQA-IMG, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.00958
  bibtex_key: zhang2025multimodaltextbook
  confidence: high
---

# vqa-benchmarks-eval

> 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining — Wenqi Zhang et al. (2025) (arXiv:2501.00958, 2025)

## What this evaluates

Evaluates vision-language models' ability to answer questions across general knowledge, OCR, mathematics, and science domains using few-shot in-context learning. It also probes the model's capacity to attend to interleaved image-text contexts through a 'cheat test' protocol.

## Datasets

- **TextVQA** — total ?; splits: test (-1)
- **OKVQA** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **MathVision** — total ?; splits: test (-1)
- **MathVerse** — total ?; splits: test (-1)
- **ScienceQA-IMG** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of total test instances.
- `perplexity` — range: other
  - Exponential of the average negative log-likelihood of the ground truth tokens.

## Input / output format

**Input**: Interleaved sequence of images and text (questions, and optionally few-shot examples with images, questions, and answers).

**Output**: Textual answer string.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Few-shot examples are randomly sampled or retrieved, which can introduce variance in accuracy scores across runs.
- The 'cheat test' protocol replaces a few-shot example with the actual test sample, measuring in-context attention rather than generalization, which may inflate scores on benchmarks requiring complex reasoning.

## Evidence (verbatim from paper)

> Following OpenFlamingo*[[5]]* and OmniCorpus*[[25]]*, we evaluate the performance of the pre-trained models on two VQA benchmarks (TextVQA*[[43]]*, OKVQA*[[35]]*), three visual reasoning benchmarks (MathVista, MathVision, MathVision), and ScienceQA-IMG*[[34]]*, covering general, OCR, mathematics, and science domains. We compute model accuracy in few-shot settings using either randomly sampled or retrieved examples as previous works*[[21], [52], [25]]*.

## Citation

```bibtex
@misc{zhang2025multimodaltextbook,
  title={2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining},
  author={Wenqi Zhang et al. (2025)},
  year={2025},
  note={arXiv:2501.00958}
}
```

- arXiv: 2501.00958