Vqa Cot Reasoning Eval

---
name: vqa-cot-reasoning-eval
description: Probes the ability of vision-language models to perform multi-step chain-of-thought reasoning on visual inputs across diverse domains like charts, documents, science diagrams, and math. It measures both direct answer accuracy and structured reasoning accuracy. Use when the user wants to benchmark on A-OKVQA, ChartQA, DocVQA, InfoVQA, TextVQA, AI2D, ScienceQA, MathVista, OCRBench, MMStar, MMMU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.16198
  bibtex_key: zhang2024improvevlmreasoning
  confidence: high
---

# vqa-cot-reasoning-eval

> Improve Vision Language Model Chain-of-thought Reasoning — Ruohong Zhang et al. (2024) (arXiv:2410.16198, 2024)

## What this evaluates

Probes the ability of vision-language models to perform multi-step chain-of-thought reasoning on visual inputs across diverse domains like charts, documents, science diagrams, and math. It measures both direct answer accuracy and structured reasoning accuracy.

## Datasets

- **A-OKVQA** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **InfoVQA** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **AI2D** — total ?; splits: test (-1)
- **ScienceQA** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Exact-match accuracy calculated as the percentage of correctly predicted answers after extracting the final answer from the model's output.

## Input / output format

**Input**: An image paired with a text prompt. Prompts are either direct questions or chain-of-thought (CoT) prompts requesting step-by-step reasoning.

**Output**: A text response. For CoT evaluation, the model must generate a reasoning trace followed by the exact pattern '# Answer: <final_answer>'.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if '# Answer:' in pred:
            pred = pred.split('# Answer:')[-1].strip()
        if pred.lower() == gold.lower():
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- For CoT evaluation, answers must be extracted specifically after the '# Answer:' pattern; failing to do so will include reasoning text in the score.
- A-OKVQA evaluation was implemented by the authors themselves, whereas other datasets use the VLMEval protocol; mixing these up causes score discrepancies.
- Direct prediction and CoT prediction should be evaluated separately, as performance varies significantly by task type (e.g., calculation tasks favor CoT, while text-rich tasks may favor direct answers).

## Evidence (verbatim from paper)

> When comparing model trained on direct only data (2) to that trained on format-aligned data (1), we observe an average gain of +5.6 in direct prediction accuracy (65.5→71.1) and a +2.9 improvement in CoT performance (62.7→65.6).

## Citation

```bibtex
@misc{zhang2024improvevlmreasoning,
  title={Improve Vision Language Model Chain-of-thought Reasoning},
  author={Ruohong Zhang et al. (2024)},
  year={2024},
  note={arXiv:2410.16198}
}
```

- arXiv: 2410.16198