Geometric Problem Solving Eval

---
name: geometric-problem-solving-eval
description: This benchmark evaluates the geometric reasoning and problem-solving capabilities of multimodal large language models. It tests whether models can accurately interpret geometric diagrams and accompanying text to produce correct final answers or select the right multiple-choice option. Use when the user wants to benchmark on GeoQA, Geometry3K, PGPS9K, MathVista-mini-GPS, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.12773
  bibtex_key: pan2025geogen
  confidence: high
---

# geometric-problem-solving-eval

> Enhancing the Geometric Problem-Solving Ability of Multimodal LLMs via Symbolic-Neural Integration — Yicheng Pan et al. (2025) (arXiv:2504.12773, 2025)

## What this evaluates

This benchmark evaluates the geometric reasoning and problem-solving capabilities of multimodal large language models. It tests whether models can accurately interpret geometric diagrams and accompanying text to produce correct final answers or select the right multiple-choice option.

## Datasets

- **GeoQA** — total 754; splits: test (754)
- **Geometry3K** — total 601; splits: test (601)
- **PGPS9K** — total 1000; splits: test (1000)
- **MathVista-mini-GPS** — total 208; splits: test (208)

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Percentage of correctly answered problems. For the Completion setting, an LLM verifier checks if the model's final numerical answer matches the ground truth. For the Choice setting, a regular expression extracts the selected option (e.g., A, B, C, D) and compares it to the ground truth.

## Input / output format

**Input**: Multimodal input consisting of a geometric diagram/image and a corresponding text-based problem statement or question.

**Output**: Either a free-form generated final answer (Completion setting) or a single letter option (A, B, C, or D) representing the selected choice (Choice setting).

## Scoring recipe

```python
def score(predictions, golds, setting='completion'):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if setting == 'completion':
            is_correct = llm_verifier_check(pred, gold)  # Checks last 3 sentences against gold
        else:
            extracted = regex_extract_option(pred)
            is_correct = (extracted == gold)
        if is_correct:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Using Top-K accuracy (e.g., Top-3 or Top-10) overestimates MLLM performance because these models can often arrive at the correct answer after multiple attempts or sampling.
- In the Choice setting, any parsing failure during option extraction is automatically counted as an incorrect prediction.
- Evaluating on the MathVista-mini subset yields high variance due to its small sample size (208 samples).

## Evidence (verbatim from paper)

> We adopt Top-1 accuracy as our primary evaluation metric, rather than Top-3 or Top-10 accuracy that symbolic or neural baselines often report. Top-K evaluation strategy considers a prediction correct if any of the Top-K answers match the ground truth. However, such a metric is less suitable for MLLMs, as these models can often reach the correct answer after multiple attempts, potentially leading to an overestimation of their actual performance. In contrast, Top-1 accuracy, which requires the model to produce the correct answer in a single response, provides a more faithful reflection of its reasoning ability.

## Citation

```bibtex
@misc{pan2025geogen,
  title={Enhancing the Geometric Problem-Solving Ability of Multimodal LLMs via Symbolic-Neural Integration},
  author={Yicheng Pan et al. (2025)},
  year={2025},
  note={arXiv:2504.12773}
}
```

- arXiv: 2504.12773