Mousi Vlm Eval

---
name: mousi-vlm-eval
description: Evaluates multimodal understanding and reasoning across visual question answering, OCR, region-level VQA, and visual conversation tasks. It probes how effectively poly-visual expert ensembles fuse information from multiple encoders compared to single-expert baselines. Use when the user wants to benchmark on LLaVA-1.5 Benchmark Suite, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.17221
  bibtex_key: fan2024mousi
  confidence: high
---

# mousi-vlm-eval

> MouSi: Poly-Visual-Expert Vision-Language Models — Xiaoran Fan et al. (2024) (arXiv:2401.17221, 2024)

## What this evaluates

Evaluates multimodal understanding and reasoning across visual question answering, OCR, region-level VQA, and visual conversation tasks. It probes how effectively poly-visual expert ensembles fuse information from multiple encoders compared to single-expert baselines.

## Datasets

- **LLaVA-1.5 Benchmark Suite** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions or selected options per benchmark, averaged across the 9 evaluation datasets.

## Input / output format

**Input**: Image and a natural language question or prompt.

**Output**: Natural language text response or multiple-choice option.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold in zip(predictions, gold):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Excluded LLaVA-Bench, VisWiz, and MME due to instability or website crashes, limiting full reproducibility.
- Performance is highly sensitive to fusion architecture (MLP vs Q-Former) and expert ordering, which vary across ablations.
- Triple-expert models hit performance ceilings likely due to limited pre-training data (665K samples), suggesting data scaling is critical.

## Evidence (verbatim from paper)

> We selected 9 of the 12 evaluation benchmarks for LLaVA-1.5 (excluding LLaVA-Bench that rely on unstable GPT4 responses, as well as VisWiz and MME for the website crashed), including VQA${}^{	ext{v2}}$; GQA; SQA${}^{	ext{I}}$ : ScienceQA-IMG; VQA${}^{	ext{T}}$: TextVQA; POPE; MMB & MMB${}^{	ext{CN}}$: MMBench & MMBench-Chinese dev results; SEED${}^{	ext{I}}$ : SEED-Bench-IMG; MM-Vet.

## Citation

```bibtex
@misc{fan2024mousi,
  title={MouSi: Poly-Visual-Expert Vision-Language Models},
  author={Xiaoran Fan et al. (2024)},
  year={2024},
  note={arXiv:2401.17221}
}
```

- arXiv: 2401.17221