Mathvista Eval

---
name: mathvista-eval
description: This benchmark evaluates the mathematical reasoning capabilities of foundation models (LLMs and LMMs) when processing visual contexts. It probes abilities such as figure interpretation, algebraic and geometric reasoning, and the integration of multimodal inputs like images, OCR text, and captions into mathematical problem-solving. Use when the user wants to benchmark on MATHVISTA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.02255
  bibtex_key: lu2023mathvista
  confidence: high
---

# mathvista-eval

> MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts — Lu et al. (2023) (arXiv:2310.02255, 2023)

## What this evaluates

This benchmark evaluates the mathematical reasoning capabilities of foundation models (LLMs and LMMs) when processing visual contexts. It probes abilities such as figure interpretation, algebraic and geometric reasoning, and the integration of multimodal inputs like images, OCR text, and captions into mathematical problem-solving.

## Datasets

- **MATHVISTA** — total ?; splits: testmini (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Accuracy is computed by comparing the normalized extracted answer against the ground truth. The evaluation pipeline extracts a short answer from the model's detailed response, normalizes it to a required format (option letter or integer), and checks for exact match.

## Input / output format

**Input**: Task description, question, choices (if applicable), and metadata, optionally accompanied by an image, generated image captions, and detected OCR text.

**Output**: A detailed response containing reasoning or code, followed by a short answer text (option letter or numerical value) that is extracted and normalized for scoring.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        extracted = extract_answer_llm(pred)
        normalized_pred = normalize(extracted)
        if normalized_pred == gold:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Using template matching or regex for answer extraction fails on long LLM responses; the protocol requires an LLM-based extractor (e.g., GPT-4) for >99.5% accuracy.
- Assuming augmented LLMs (with OCR/captions) can match native LMMs, as they lack direct visual perception and struggle with complex figure interpretation and shape detection.
- Reporting results on the full test set when the paper explicitly provides and highlights scores only on the 'testmini' subset.

## Evidence (verbatim from paper)

> Finally, the extracted answer is normalized to a required answer format (e.g., an option letter or an integer), and the target metric scores are computed. Taking advantage of the fact that the instances in MATHVISTA are either multiple-choice questions for textual answers or free-form questions for numerical answers, accuracy scores are used as metrics for deterministic evaluation.

## Citation

```bibtex
@misc{lu2023mathvista,
  title={MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts},
  author={Lu et al. (2023)},
  year={2023},
  note={arXiv:2310.02255}
}
```

- arXiv: 2310.02255