Mova Eval

---
name: mova-eval
description: Evaluates multimodal large language models' capabilities across general visual question answering, text-oriented VQA (charts, documents, diagrams), visual grounding (referring expression comprehension), and specialized medical VQA. It also assesses general multimodal reasoning and hallucination resistance. Use when the user wants to benchmark on MME, MMBench, MMBench-CN, QBench, MathVista, MathVerse, POPE, VQAv2, GQA, SQA-I, TextVQA, ChartQA, DocVQA, AI2D, RefCOCO, RefCOCO+, RefCOCOg, VQA-RAD, SLAKE, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.13046
  bibtex_key: zong2024mova
  confidence: high
---

# mova-eval

> MoVA: Adapting Mixture of Vision Experts to Multimodal Context — Zong et al. (2024) (arXiv:2404.13046, 2024)

## What this evaluates

Evaluates multimodal large language models' capabilities across general visual question answering, text-oriented VQA (charts, documents, diagrams), visual grounding (referring expression comprehension), and specialized medical VQA. It also assesses general multimodal reasoning and hallucination resistance.

## Datasets

- **MME** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **MMBench-CN** — total ?; splits: test (-1)
- **QBench** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **MathVerse** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **VQAv2** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **SQA-I** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **AI2D** — total ?; splits: test (-1)
- **RefCOCO** — total ?; splits: val (-1), test-A (-1), test-B (-1)
- **RefCOCO+** — total ?; splits: val (-1), test-A (-1), test-B (-1)
- **RefCOCOg** — total ?; splits: val (-1), test (-1)
- **VQA-RAD** — total ?; splits: Open (-1), Close (-1)
- **SLAKE** — total ?; splits: Open (-1), Close (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions or correctly localized bounding boxes. Calculated as (number of correct predictions / total number of samples) * 100.

## Input / output format

**Input**: Paired image and text instruction/question.

**Output**: Text answer for VQA tasks; bounding box coordinates for Referring Expression Comprehension (REC) tasks.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
    return (correct / len(gold)) * 100
```

## Common pitfalls

- RefCOCO benchmarks use multiple distinct splits (val, test-A, test-B) that must be reported separately.
- Medical VQA datasets (VQA-RAD, SLAKE) have both 'Open' and 'Close' settings that yield different scores.
- Some baselines are evaluated zero-shot (marked with *) while others are fine-tuned, requiring careful comparison.

## Evidence (verbatim from paper)

> We develop 105 human-verified testing samples that should be answered using novel experts for the expert routing task. These novel experts encompass 7 vision models [93, 71, 94, 91, 95, 54, 96] on various computer vision tasks and each expert corresponds to 15 evaluation samples. We manually check the correctness of the expert routing result. As presented in Table 10, a lightweight network, such as a MLP classifier fails to generalize to this open-world setting.

## Citation

```bibtex
@misc{zong2024mova,
  title={MoVA: Adapting Mixture of Vision Experts to Multimodal Context},
  author={Zong et al. (2024)},
  year={2024},
  note={arXiv:2404.13046}
}
```

- arXiv: 2404.13046