Multimodal Vqa Eval

---
name: multimodal-vqa-eval
description: Evaluates the zero-shot and few-shot visual question answering capabilities of multimodal large language models (MLLMs). It probes scene and spatial understanding, OCR capabilities, commonsense knowledge reasoning, and multimodal in-context learning across diverse benchmarks. Use when the user wants to benchmark on GQA, VQA-v2, VizWiz, TextVQA, OKVQA, POPE, MMMU (Val), MMBench (Dev), MMStar, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.15162
  bibtex_key: wang2025unifilter
  confidence: high
---

# multimodal-vqa-eval

> Train a Unified Multimodal Data Quality Classifier with Synthetic Data — Wang et al. (2025) (arXiv:2510.15162, 2025)

## What this evaluates

Evaluates the zero-shot and few-shot visual question answering capabilities of multimodal large language models (MLLMs). It probes scene and spatial understanding, OCR capabilities, commonsense knowledge reasoning, and multimodal in-context learning across diverse benchmarks.

## Datasets

- **GQA** — total ?; splits: test (-1)
- **VQA-v2** — total ?; splits: test (-1)
- **VizWiz** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **OKVQA** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **MMMU (Val)** — total ?; splits: val (-1)
- **MMBench (Dev)** — total ?; splits: dev (-1)
- **MMStar** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Computed as the mean score over 5 random seeds for demonstration example sampling in few-shot settings.

## Input / output format

**Input**: Image(s) and text prompt (question or few-shot demonstrations + question).

**Output**: Text answer to the visual question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Few-shot results are averaged over 5 random seeds for demonstration sampling, not a single deterministic run.
- VQA-v2 scores may be inflated for baselines like DFN because MSCOCO (used to train DFN) overlaps with VQA-v2's source data.
- Zero-shot and few-shot evaluations measure different capabilities (direct reasoning vs. in-context learning), so they should not be directly compared without context.

## Evidence (verbatim from paper)

> Finally, the UniFilter induced MLLM achieves +0.7 and +2.8 average accuracy improvements over the DFN baseline on 4-shot and 8-shot in-context learning, respectively.

## Citation

```bibtex
@misc{wang2025unifilter,
  title={Train a Unified Multimodal Data Quality Classifier with Synthetic Data},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.15162}
}
```

- arXiv: 2510.15162