Gsm8k V Eval

---
name: gsm8k-v-eval
description: This benchmark evaluates vision-language models' ability to perform multi-step mathematical reasoning using purely visual, comic-style narratives instead of text. It specifically probes challenges in inter-image semantic understanding, object grounding, and extracting numerical relationships from multi-panel visual contexts. Use when the user wants to benchmark on GSM8K-V, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.25160
  bibtex_key: yuan2025gsm8kv
  confidence: high
---

# gsm8k-v-eval

> GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts — Fan Yuan et al. (2025) (arXiv:2509.25160, 2025)

## What this evaluates

This benchmark evaluates vision-language models' ability to perform multi-step mathematical reasoning using purely visual, comic-style narratives instead of text. It specifically probes challenges in inter-image semantic understanding, object grounding, and extracting numerical relationships from multi-panel visual contexts.

## Datasets

- **GSM8K-V** — total ?; splits: test (-1); repo https://github.com/ZJU-REAL/GSM8K-V

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of problems where the model's final predicted numeric answer exactly matches the ground truth answer. Calculated as (number of correct predictions / total number of problems) * 100.

## Input / output format

**Input**: Multi-image comic-style visual narratives depicting grade school math word problems. Models receive a sequence of images representing the problem context and are prompted to reason step by step.

**Output**: Only the final numeric answer in the required format (integer, decimal, or fraction), without additional reasoning text.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # Normalize whitespace and numeric formats
        pred_val = normalize_numeric(pred.strip())
        gold_val = normalize_numeric(gold.strip())
        if pred_val == gold_val:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Models often fail to maintain semantic consistency across multiple comic panels, leading to incorrect intermediate values or misaligned object grounding.
- Strict output formatting is required; models that output reasoning text alongside the answer may cause parsing failures if not explicitly stripped before scoring.
- Visual ambiguity in comic-style representations can cause models to misidentify quantities or relationships, resulting in systematic category-specific drops (e.g., 'Other' category).

## Evidence (verbatim from paper)

> For GSM8K-V, models are instructed to reason step by step over the images and report only the final numeric answer in the required format (integer, decimal, or fraction). The best-performing model, Gemini-2.5-Pro, achieved only 46.93% accuracy, while other models—including flagship ones such as GPT-5 and Llama-4-17B-128E—reached merely around 30%.

## Citation

```bibtex
@misc{yuan2025gsm8kv,
  title={GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts},
  author={Fan Yuan et al. (2025)},
  year={2025},
  note={arXiv:2509.25160}
}
```

- arXiv: 2509.25160