Deepvision 103k Eval

---
name: deepvision-103k-eval
description: Evaluates the multimodal mathematical reasoning and general multimodal reasoning capabilities of vision-language models. It probes visual perception, step-by-step logical deduction, and cross-domain generalization on K12-level math and broader visual tasks. Use when the user wants to benchmark on Multimodal Math & General Reasoning Benchmarks (WeMath, MathVerse_vision, MathVision, LogicVista, MMMU_VAL, MMMU_Pro_full, M^3CoT), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.16742
  bibtex_key: sun2026deepvision103k
  confidence: high
---

# deepvision-103k-eval

> DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoning — Sun et al. (2026) (arXiv:2602.16742, 2026)

## What this evaluates

Evaluates the multimodal mathematical reasoning and general multimodal reasoning capabilities of vision-language models. It probes visual perception, step-by-step logical deduction, and cross-domain generalization on K12-level math and broader visual tasks.

## Datasets

- **Multimodal Math & General Reasoning Benchmarks (WeMath, MathVerse_vision, MathVision, LogicVista, MMMU_VAL, MMMU_Pro_full, M^3CoT)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Calculated as the number of predictions matching the ground truth answer divided by the total number of evaluation instances, multiplied by 100.

## Input / output format

**Input**: Multimodal inputs consisting of mathematical problem images paired with text prompts/questions.

**Output**: Textual responses containing the final answer and/or step-by-step reasoning.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = (correct / len(predictions)) * 100
```

## Common pitfalls

- Confusing the training reward signal (rule-based +1/0 for answer correctness) with the evaluation metric.
- Failing to use the official decoding parameters and 32K max token limit, which significantly impacts performance on these benchmarks.
- Not accounting for varying answer formats across benchmarks (e.g., multiple-choice vs. open-ended) before applying exact-match scoring.

## Evidence (verbatim from paper)

> We evaluated our models on the following benchmarks: (1) Multimodal Math: WeMath, MathVerse_vision, MathVision, and LogicVista. (2) General Multimodal: MMMU_VAL, MMMU_Pro_full and M^3CoT. For inference parameters, we set the maximum token length at 32K for all evaluation. Decoding parameters follow the official recommendations. ... Qwen3-VL-8B-DeepVision reaches sota results on WeMath (85.11%), MiMo-VL-7B-DeepVision reaches sota results on LogicVista (65.62%.

## Citation

```bibtex
@misc{sun2026deepvision103k,
  title={DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoning},
  author={Sun et al. (2026)},
  year={2026},
  note={arXiv:2602.16742}
}
```

- arXiv: 2602.16742