We Math Eval

---
name: we-math-eval
description: Evaluates the visual mathematical reasoning capabilities of Large Multimodal Models (LMMs). It probes their ability to decompose composite problems, apply hierarchical knowledge concepts, and reason through multi-step visual math tasks without relying on rote memorization. Use when the user wants to benchmark on We-Math testmini, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.01284
  bibtex_key: qiao2024wemath
  confidence: high
---

# we-math-eval

> We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? — Qiao et al. (2024) (arXiv:2407.01284, 2024)

## What this evaluates

Evaluates the visual mathematical reasoning capabilities of Large Multimodal Models (LMMs). It probes their ability to decompose composite problems, apply hierarchical knowledge concepts, and reason through multi-step visual math tasks without relying on rote memorization.

## Datasets

- **We-Math testmini** — total 1740; splits: testmini (1740); repo https://github.com/We-Math/We-Math

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted options after regex matching against ground truth.
- `four-dimensional metric (IK, IG, CM, RM)` — range: percent
  - Classifies model reasoning into Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), or Rote Memorization (RM). Reported as percentages under strict and loose evaluation settings.

## Input / output format

**Input**: Multiple-choice visual mathematical problems accompanied by an extra "uncertain" option to mitigate option-deduction.

**Output**: Predicted option letter or answer string.

## Scoring recipe

```python
# Accuracy
pred = extract_regex(model_output)
score = 1.0 if pred == ground_truth else 0.0

# Four-dimensional metric
# Classify reasoning trace into:
# IK: Insufficient Knowledge (fails to identify core concept)
# IG: Inadequate Generalization (knows concept but fails application)
# CM: Complete Mastery (correct reasoning & answer)
# RM: Rote Memorization (matches training without reasoning)
# Compute category percentages under strict/loose settings.
```

## Common pitfalls

- Models often bypass reasoning by deducing answers directly from multiple-choice options.
- Performance drops sharply as problem complexity (number of knowledge concepts/steps) increases.
- Visual measurement tasks (e.g., angles, units) are disproportionately difficult due to visual encoder limitations.

## Evidence (verbatim from paper)

> For automated evaluation, we standardize all samples into a multiple-choice format. We use regex to match the LMMs’ predictions and then calculate their accuracy against the ground-truth answers for main results. For analyses in section [3.2] and [3.3], we utilize the four-dimensional metric described in section [2.2] for assessment.

## Citation

```bibtex
@misc{qiao2024wemath,
  title={We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?},
  author={Qiao et al. (2024)},
  year={2024},
  note={arXiv:2407.01284}
}
```

- arXiv: 2407.01284