Wan Image Understanding Eval

---
name: wan-image-understanding-eval
description: Evaluates the model's multimodal and text understanding capabilities across a suite of standard academic benchmarks covering visual question answering, reasoning, hallucination detection, and text-based reasoning. Use when the user wants to benchmark on MMMU, MMStar, MathVista, HalluBench, MMBench, OCRBench, AI2D, AIME, GPQA, HLE, LCBV6, or asks about evaluating this task. Reports average score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.19858
  bibtex_key: mao2026wanimage
  confidence: high
---

# wan-image-understanding-eval

> Wan-Image: Pushing the Boundaries of Generative Visual Intelligence — Mao et al. (2026) (arXiv:2604.19858, 2026)

## What this evaluates

Evaluates the model's multimodal and text understanding capabilities across a suite of standard academic benchmarks covering visual question answering, reasoning, hallucination detection, and text-based reasoning.

## Datasets

- **MMMU** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **HalluBench** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **AI2D** — total ?; splits: test (-1)
- **AIME** — total ?; splits: test (-1)
- **GPQA** — total ?; splits: test (-1)
- **HLE** — total ?; splits: test (-1)
- **LCBV6** — total ?; splits: test (-1)

## Metrics

- `average score` **(primary)** — range: [0, 100]
  - Percentage of correctly answered questions or normalized benchmark score as reported by the official benchmark leaderboards. Averages are computed across selected benchmarks.

## Input / output format

**Input**: Image and/or text prompts/questions provided by the benchmark.

**Output**: Textual answers or multiple-choice selections.

## Scoring recipe

```python
def compute_average_score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100

scores = [compute_average_score(preds, golds) for preds, golds in dataset_splits]
avg_score = sum(scores) / len(scores)
```

## Common pitfalls

- Scores are reported as averages across benchmarks rather than per-dataset in the main text.
- The table uses '–' for missing indicators, which should be excluded from averages.
- Comparison includes both understanding-only and unified models, so direct comparison requires matching model scale/type.

## Evidence (verbatim from paper)

> For Image Understanding, we selected 7 classical multi-modal benchmarks to assess the model’s performance, including MMMU, MMStar, MathVista, HalluBench, MMBench, OCRBench, and AI2D. The results shown in Table 4 demonstrate that our model exhibits superior instruct and thinking capabilities: in the instruct mode, it achieves an average score 1.6 points higher than the baseline and remains comparable to other open-source understanding-only instruct models. Concurrently, in thinking mode, our model outperforms the baseline by 5.4 points and exceeds the performance of other open-source understanding-only thinking models. Compared to other unified models, our approach maintains a distinct and absolute lead. Regarding Text Understanding, our model demonstrates significant improvements over the showcased understanding-only models in both instruct and thinking modes on four common benchmarks: AIME, GPQA, HLE, and LCBV6.

## Citation

```bibtex
@misc{mao2026wanimage,
  title={Wan-Image: Pushing the Boundaries of Generative Visual Intelligence},
  author={Mao et al. (2026)},
  year={2026},
  note={arXiv:2604.19858}
}
```

- arXiv: 2604.19858