Mammoth Vl Multimodal Eval

---
name: mammoth-vl-multimodal-eval
description: Evaluates multimodal reasoning and instruction-following capabilities across single-image, multi-image, and video scenarios. Probes OCR, chart/document understanding, mathematical reasoning, and real-world visual interactions. Use when the user wants to benchmark on AI2D, ChartQA, DocVQA, InfoVQA, MMStar, MMMU, MMMU-Pro, SeedBench, MMBench, MMvet, Mathverse, Mathvista, RealworldQA, WildVision, Llava-Wilder-Small, MuirBench, MEGABench, EgoSchema, PerceptionTest, SeedBench (Video), MLVU, MVBench, VideoMME, or asks about evaluating this task. Reports Benchmark Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.05237
  bibtex_key: guo2024mammothvl
  confidence: high
---

# mammoth-vl-multimodal-eval

> MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale — Jarvis Guo et al. (2024) (arXiv:2412.05237, 2024)

## What this evaluates

Evaluates multimodal reasoning and instruction-following capabilities across single-image, multi-image, and video scenarios. Probes OCR, chart/document understanding, mathematical reasoning, and real-world visual interactions.

## Datasets

- **AI2D** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **InfoVQA** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MMMU-Pro** — total ?; splits: test (-1)
- **SeedBench** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **MMvet** — total ?; splits: test (-1)
- **Mathverse** — total ?; splits: test (-1)
- **Mathvista** — total ?; splits: test (-1)
- **RealworldQA** — total ?; splits: test (-1)
- **WildVision** — total ?; splits: test (-1)
- **Llava-Wilder-Small** — total ?; splits: test (-1)
- **MuirBench** — total ?; splits: test (-1)
- **MEGABench** — total ?; splits: test (-1)
- **EgoSchema** — total ?; splits: test (-1)
- **PerceptionTest** — total ?; splits: test (-1)
- **SeedBench (Video)** — total ?; splits: video (-1)
- **MLVU** — total ?; splits: test (-1)
- **MVBench** — total ?; splits: test (-1)
- **VideoMME** — total ?; splits: w/o subs (-1)

## Metrics

- `Benchmark Score` **(primary)** — range: percent
  - Percentage score reported per benchmark. Varies by dataset (e.g., exact match for QA, F1 for VQA, or custom scoring for reasoning). All results are aggregated and reported as percentages in the tables.

## Input / output format

**Input**: Image(s) or video frames paired with natural language instructions/prompts.

**Output**: Text response generated via greedy decoding.

## Scoring recipe

```python
# Pseudo-code for evaluation protocol
for benchmark in benchmarks:
    model_output = model.generate(prompt, image, greedy=True, zero_shot=True)
    score = benchmark.evaluate(model_output, gold)  # Uses benchmark-specific metric
    results[benchmark] = score * 100  # Reported as percentage
return results
```

## Common pitfalls

- Using non-zero-shot or non-greedy decoding settings, which deviates from the reported evaluation protocol.
- Assuming a single uniform metric across all 23 benchmarks; each benchmark uses its own specific scoring method.
- Ignoring split variations (e.g., 'dev' vs 'test' vs 'w/o subs') explicitly noted in the benchmark tables.

## Evidence (verbatim from paper)

> We conduct standardized, reproducible evaluations of our model across all 23 benchmarks using LMMs-Eval(Zhang et al., [2024a]). To ensure a fair comparison with other MLLMs, we primarily report results from the original papers. When results are unavailable, we onboard the models in LMMs-Eval and evaluate them using consistent settings. All results are reported using greedy decoding and zero-shot settings unless specified.

## Citation

```bibtex
@misc{guo2024mammothvl,
  title={MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale},
  author={Jarvis Guo et al. (2024)},
  year={2024},
  note={arXiv:2412.05237}
}
```

- arXiv: 2412.05237