Mulberry Eval

---
name: mulberry-eval
description: Evaluates the multimodal reasoning and understanding capabilities of MLLMs across diverse domains including mathematics, chart interpretation, scientific/medical images, and hallucination detection. It measures how well models generate step-by-step reasoning paths and reflect on errors to produce correct answers. Use when the user wants to benchmark on MathVista, MMStar, MMMU, ChartQA, DynaMath, HallBench, MM-Math, MMEsum, or asks about evaluating this task. Reports Average Benchmark Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.18319
  bibtex_key: yao2024mulberry
  confidence: high
---

# mulberry-eval

> Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search — Yao et al. (2024) (arXiv:2412.18319, 2024)

## What this evaluates

Evaluates the multimodal reasoning and understanding capabilities of MLLMs across diverse domains including mathematics, chart interpretation, scientific/medical images, and hallucination detection. It measures how well models generate step-by-step reasoning paths and reflect on errors to produce correct answers.

## Datasets

- **MathVista** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **DynaMath** — total ?; splits: test (-1)
- **HallBench** — total ?; splits: test (-1)
- **MM-Math** — total ?; splits: test (-1)
- **MMEsum** — total ?; splits: test (-1)

## Metrics

- `Average Benchmark Score` **(primary)** — range: percent
  - Arithmetic mean of the accuracy (or normalized score) across the 8 listed benchmarks. For MMEsum, the raw sum of sub-task scores is used directly in the average.
- `MathVista` — range: percent
  - Accuracy on multimodal math problems.

## Input / output format

**Input**: A text task instruction paired with an input image.

**Output**: A step-by-step reasoning path followed by a final answer.

## Scoring recipe

```python
For each of the 8 benchmarks, compute the accuracy (or retrieve the provided aggregate score).
Calculate the arithmetic mean of these 8 scores to report the AVG column.
Note: MMEsum uses a raw score scale (sum of sub-task scores) rather than percentage, so it should be treated as-is when averaging.
```

## Common pitfalls

- MMEsum reports a raw sum score (e.g., ~2000) rather than a percentage, so it should not be averaged directly with percentage-based benchmarks without normalization.
- Several baseline models have missing values ('-') in the table for certain datasets, which may skew the reported average if not handled correctly.
- The paper evaluates reasoning models but does not specify a strict format for extracting the final answer from the reasoning trace, relying on standard benchmark parsers.

## Evidence (verbatim from paper)

> It shows that, using the same base model LLaVA-NeXT-8B (Li et al., [2024]), our Mulberry outperforms LLaVA-Reasoner-8B and Insight-V-8B by +5.7% and +6.5% on mathematical benchmark MathVista, and by +3.0% and +1.0% on multi-disciplinary benchmark MMMU, respectively.

## Citation

```bibtex
@misc{yao2024mulberry,
  title={Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search},
  author={Yao et al. (2024)},
  year={2024},
  note={arXiv:2412.18319}
}
```

- arXiv: 2412.18319