Mm Upt Eval

---
name: mm-upt-eval
description: Evaluates the multi-modal mathematical reasoning capabilities of MLLMs on diverse visual math problems including geometry, charts, and tables. It tests the model's ability to solve multiple-choice and fill-in-the-blank questions using both human-created and synthetically generated unlabeled data. Use when the user wants to benchmark on MathVision, MathVerse, MathVista, We-Math, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.22453
  bibtex_key: wei2025mmupt
  confidence: high
---

# mm-upt-eval

> First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training — Lai Wei et al. (2025) (arXiv:2505.22453, 2025)

## What this evaluates

Evaluates the multi-modal mathematical reasoning capabilities of MLLMs on diverse visual math problems including geometry, charts, and tables. It tests the model's ability to solve multiple-choice and fill-in-the-blank questions using both human-created and synthetically generated unlabeled data.

## Datasets

- **MathVision** — total ?; splits: test (-1)
- **MathVerse** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **We-Math** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions in the benchmark.

## Input / output format

**Input**: Multi-modal mathematical reasoning problems containing images and text (geometry diagrams, charts, tables, multiple-choice, or fill-in-the-blank questions).

**Output**: Textual answer or solution to the mathematical problem.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- The paper evaluates on multiple benchmarks but reports average accuracy across them, which can mask per-benchmark performance variations.
- Training datasets (Geometry3K, GeoQA, MMR1) are used without ground-truth labels during unsupervised post-training, so evaluation relies on external benchmark labels not present during training.
- Synthetic data generation strategies (In-Context vs Direct) are compared, but the evaluation metric remains standard accuracy without checking for hallucination or factual correctness of generated questions.

## Evidence (verbatim from paper)

> We report accuracy (%) for each method on MathVision, MathVerse, MathVista, and We-Math.

## Citation

```bibtex
@misc{wei2025mmupt,
  title={First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training},
  author={Lai Wei et al. (2025)},
  year={2025},
  note={arXiv:2505.22453}
}
```

- arXiv: 2505.22453