Math Vision Eval

---
name: math-vision-eval
description: Evaluates multimodal mathematical reasoning capabilities of LLMs and LMMs on problems with visual contexts. Probes geometric invariance, spatial reasoning, and deep mathematical reasoning across 16 disciplines and 5 difficulty levels. Use when the user wants to benchmark on MATH-V, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.14804
  bibtex_key: wang2024mathvision
  confidence: high
---

# math-vision-eval

> Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset — Wang et al. (2024) (arXiv:2402.14804, 2024)

## What this evaluates

Evaluates multimodal mathematical reasoning capabilities of LLMs and LMMs on problems with visual contexts. Probes geometric invariance, spatial reasoning, and deep mathematical reasoning across 16 disciplines and 5 difficulty levels.

## Datasets

- **MATH-V** — total 3040; splits: testmini (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions in the dataset.

## Input / output format

**Input**: A mathematical problem presented with an accompanying image (visual context), provided in a zero-shot or Chain-of-Thought prompt format.

**Output**: A final answer to the mathematical problem, either as a multiple-choice selection or an open-ended solution, depending on the question type.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Chain-of-Thought prompting does not consistently improve performance and can actually decrease accuracy for some models (e.g., Gemini Pro, Qwen-VL-Max).
- Performance varies significantly across mathematical subjects (e.g., struggles in counting vs. arithmetic), so reporting only overall accuracy masks domain-specific weaknesses.
- Human baseline was evaluated on the 'testmini' subset only, which may not represent the full dataset difficulty.

## Evidence (verbatim from paper)

> Notably, the overall accuracy for the leading model, GPT-4V, is only 22.76%, which, while being the highest, still indicates a significant challenge in tackling the dataset.

## Citation

```bibtex
@misc{wang2024mathvision,
  title={Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2402.14804}
}
```

- arXiv: 2402.14804