Jmmmu Pro Eval

---
name: jmmmu-pro-eval
description: Evaluates multimodal language models' ability to perform integrated visual-textual reasoning on Japanese-language tasks where questions and reference images are combined into a single composite image. It specifically probes OCR capabilities, visual perception, and cross-modal alignment in a multilingual context. Use when the user wants to benchmark on JMMMU-Pro, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.14620
  bibtex_key: miyai2025jmmmu
  confidence: high
---

# jmmmu-pro-eval

> JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction — Miyai et al. (2025) (arXiv:2512.14620, 2025)

## What this evaluates

Evaluates multimodal language models' ability to perform integrated visual-textual reasoning on Japanese-language tasks where questions and reference images are combined into a single composite image. It specifically probes OCR capabilities, visual perception, and cross-modal alignment in a multilingual context.

## Datasets

- **JMMMU-Pro** — total 1320; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard exact-match accuracy: the proportion of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total predictions) × 100.

## Input / output format

**Input**: A single composite image containing both the reference visual material and the Japanese question text, requiring the model to perform joint visual perception and OCR before answering.

**Output**: A natural language text response containing the final answer to the question.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Open-source models are evaluated with both Direct and Chain-of-Thought prompts, reporting the higher score, while closed-source models only use Direct prompts, creating an asymmetric evaluation setup.
- The composite image format requires strong OCR and visual perception; poor performance may stem from text extraction failures rather than reasoning deficits.
- Temperature is fixed at 0 for open-source models but uses default settings for closed-source models, which can affect output consistency and reproducibility.

## Evidence (verbatim from paper)

> Following MMMU-Pro [59], we evaluate the open-source LMMs with both Direct and CoT prompts (as shown in Section C), and report the higher ones in the overall results. For the closed-source LMMs, they perform reasoning regardless of the prompt types, so we evaluated them using only the Direct Prompt. ... As shown in Table 1, most open-source LMMs, except for Qwen2.5-VL-7B, show a substantial decline in accuracy on JMMMU-Pro relative to JMMMU.

## Citation

```bibtex
@misc{miyai2025jmmmu,
  title={JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction},
  author={Miyai et al. (2025)},
  year={2025},
  note={arXiv:2512.14620}
}
```

- arXiv: 2512.14620