Mathcoder Vl Eval

---
name: mathcoder-vl-eval
description: Evaluates multimodal mathematical reasoning capabilities across diverse benchmarks, focusing on geometry problem solving, multi-step reasoning, and cross-modal alignment between visual diagrams and mathematical concepts. Use when the user wants to benchmark on MATH-Vision, MathVista, MathVerse, GAOKAO-MM, We-Math, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.10557
  bibtex_key: wang2025mathcodervl
  confidence: high
---

# mathcoder-vl-eval

> MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning — Wang et al. (2025) (arXiv:2505.10557, 2025)

## What this evaluates

Evaluates multimodal mathematical reasoning capabilities across diverse benchmarks, focusing on geometry problem solving, multi-step reasoning, and cross-modal alignment between visual diagrams and mathematical concepts.

## Datasets

- **MATH-Vision** — total 3040; splits: test (3040)
- **MathVista** — total ?; splits: test (-1)
- **MathVerse** — total ?; splits: test (-1)
- **GAOKAO-MM** — total ?; splits: test (-1)
- **We-Math** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the percentage of correctly answered problems out of the total number of evaluated instances. Standard exact-match or semantic-match accuracy is used per benchmark conventions.

## Input / output format

**Input**: Multimodal instances containing mathematical problem text and associated images/diagrams (e.g., geometric figures, charts, or visual contexts).

**Output**: Text-based answers or step-by-step solutions generated via greedy decoding.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- MathVista evaluation must be restricted to the Geometry Problem Solving (GPS) subset, as other tasks prioritize natural image recognition over mathematical reasoning.
- Performance varies significantly by input modality (Text-Dominant vs Vision-Only), so reporting aggregate scores without modality breakdowns can be misleading.
- Multi-step problem accuracy requires explicit stratification by step count (1-step, 2-step, 3-step) to properly assess chain-of-thought reasoning capabilities.

## Evidence (verbatim from paper)

> Benchmarks. We assess our models across a diverse set of widely recognized mathematical benchmarks. The MATH-Vision dataset includes 3,040 visually contextualized math problems sourced from real-world competitions. MathVista is a well-known dataset designed for evaluating reasoning in visual contexts. Many tasks in MathVista require more emphasis on natural image recognition rather than math reasoning abilities, so we only report results on the Geometry Problem Solving (GPS) subset. Overall Performances. As shown in Table[1], MathCoder-VL demonstrates strong performance across multiple mathematical benchmarks... MathCoder-VL-8B achieves the highest accuracy among open-source LMMs of similar sizes, with 26.1% on MATH-Vision, 46.5% on MathVerse, and an impressive 73.6% on the MathVista (GPS).

## Citation

```bibtex
@misc{wang2025mathcodervl,
  title={MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.10557}
}
```

- arXiv: 2505.10557