Mathverse Eval

---
name: mathverse-eval
description: This benchmark evaluates the visual mathematical reasoning capabilities of multi-modal large language models (MLLMs), specifically probing whether they genuinely interpret geometric diagrams or merely rely on textual redundancy. It measures performance across different problem formulations (varying text/image ratios) and mathematical subjects like plane geometry, solid geometry, and functions. Use when the user wants to benchmark on MATHVERSE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.14624
  bibtex_key: zhang2024mathverse
  confidence: high
---

# mathverse-eval

> MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? — Renrui Zhang et al. (2024) (arXiv:2403.14624, 2024)

## What this evaluates

This benchmark evaluates the visual mathematical reasoning capabilities of multi-modal large language models (MLLMs), specifically probing whether they genuinely interpret geometric diagrams or merely rely on textual redundancy. It measures performance across different problem formulations (varying text/image ratios) and mathematical subjects like plane geometry, solid geometry, and functions.

## Datasets

- **MATHVERSE** — total 15672; splits: testmini (4728), test (10944)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Binary correctness of the final predicted answer against the ground truth, averaged across all instances. Reported as a percentage.
- `CoT evaluation score` — range: [0, 1]
  - Scores intermediate reasoning steps (0 or 1 per step). Multi-step score is the average of step scores. Final score combines multi-step score (weight 0.7) and final answer correctness (weight 0.3).

## Input / output format

**Input**: Multi-modal input consisting of a mathematical diagram image and a corresponding text question. The benchmark includes six problem versions that manipulate the presence of text and visual elements (e.g., Text Dominant, Text Only, Vision-only).

**Output**: Model generates a step-by-step Chain-of-Thought reasoning process followed by a final answer (multiple-choice selection or free-form text).

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

def compute_cot_score(steps, final_preds, golds):
    step_scores = [1 if step_correct else 0 for step_correct in steps]
    multi_step = sum(step_scores) / len(step_scores)
    final_correct = sum(1 for p, g in zip(final_preds, golds) if p == g)
    final_score = final_correct / len(golds)
    return multi_step * 0.7 + final_score * 0.3
```

## Common pitfalls

- Models often achieve high scores by exploiting textual redundancy in the problem descriptions rather than actually interpreting the diagram, leading to inflated performance when visual input is removed.
- The CoT evaluation score can mask final answer errors by rewarding correct intermediate reasoning steps, creating a significant discrepancy between CoT-E and standard accuracy metrics.
- Evaluations are strictly zero-shot; using few-shot prompting or fine-tuning on the benchmark will invalidate the reported baseline comparisons.

## Evidence (verbatim from paper)

> In the subsequent experiments, all quantitative results are assessed using the testmini subset of MATHVERSE. ... The highest accuracy for closed-source and open-source MLLMs is marked in red and blue respectively.

## Citation

```bibtex
@misc{zhang2024mathverse,
  title={MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?},
  author={Renrui Zhang et al. (2024)},
  year={2024},
  note={arXiv:2403.14624}
}
```

- arXiv: 2403.14624