Multimodal Understanding Eval

---
name: multimodal_understanding_eval
description: Evaluates a model's ability to understand and reason over diverse visual inputs, including general VQA, document/chart understanding, OCR, and hallucination robustness. Use when the user wants to benchmark on MMMU(Val), MMStar, MME, OCRBench, HallB(Avg), MMB(Dev En V1.1), TextVQA, DoCVQA, InfoVQA, AI2D, ChartQA, RWQA, or asks about evaluating this task. Reports VLMEvalKit score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.17520
  bibtex_key: yang2025instructvla
  confidence: high
---

# multimodal_understanding_eval

> InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation — Yang et al. (2025) (arXiv:2507.17520, 2025)

## What this evaluates

Evaluates a model's ability to understand and reason over diverse visual inputs, including general VQA, document/chart understanding, OCR, and hallucination robustness.

## Datasets

- **MMMU(Val)** — total ?; splits: val (-1)
- **MMStar** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **HallB(Avg)** — total ?; splits: test (-1)
- **MMB(Dev En V1.1)** — total ?; splits: dev (-1)
- **TextVQA** — total ?; splits: test (-1)
- **DoCVQA** — total ?; splits: test (-1)
- **InfoVQA** — total ?; splits: test (-1)
- **AI2D** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **RWQA** — total ?; splits: test (-1)

## Metrics

- `VLMEvalKit score` **(primary)** — range: percent
  - Percentage of correctly answered questions or matched options, computed automatically via VLMEvalKit.

## Input / output format

**Input**: Image(s) and text prompt/question.

**Output**: Text response (answer or reasoning).

## Scoring recipe

```python
predictions, gold = load_data(benchmark)
score = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold) * 100
return score
```

## Common pitfalls

- Benchmarks are evaluated using VLMEvalKit, so manual scoring may differ from reported numbers.
- Some benchmarks like MME and HallB report multiple sub-scores; the paper uses averaged or specific sub-scores (e.g., HallB Avg).

## Evidence (verbatim from paper)

> We use the automatic evaluation from VLMEvalKit [65] including MMMU(Val) [66], MMStar [67], MME [68], OCRBench [69], HallB(Avg) [70], MMB(Dev En V1.1) [71], TextVQA [72], DoCVQA [73], InfoVQA [74], AI2D [75], ChartQA [76] and RWQA [77].

## Citation

```bibtex
@misc{yang2025instructvla,
  title={InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2507.17520}
}
```

- arXiv: 2507.17520