Blip3 Multimodal Benchmarks Eval

---
name: blip3-multimodal-benchmarks-eval
description: Evaluates large multimodal models on single- and multi-image understanding, covering general VQA, domain knowledge, OCR, hallucination, and interleaved image-text reasoning. Use when the user wants to benchmark on SEED-IMG, MMB(dev), MMStar, MME(norm), RWQA, MMVet, MMMU(val), MathVista, TextVQA, OCRBench, POPE, HalBench, BLINK, QBench, MuirBench, Mantis-Eval, or asks about evaluating this task. Reports benchmark score, average score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.08872
  bibtex_key: xue2024xgenmm
  confidence: high
---

# blip3-multimodal-benchmarks-eval

> xGen-MM (BLIP-3): A Family of Open Large Multimodal Models — Le Xue et al. (2024) (arXiv:2408.08872, 2024)

## What this evaluates

Evaluates large multimodal models on single- and multi-image understanding, covering general VQA, domain knowledge, OCR, hallucination, and interleaved image-text reasoning.

## Datasets

- **SEED-IMG** — total ?; splits: test (-1)
- **MMB(dev)** — total ?; splits: dev (-1)
- **MMStar** — total ?; splits: test (-1)
- **MME(norm)** — total ?; splits: test (-1)
- **RWQA** — total ?; splits: test (-1)
- **MMVet** — total ?; splits: test (-1)
- **MMMU(val)** — total ?; splits: val (-1)
- **MathVista** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **HalBench** — total ?; splits: test (-1)
- **BLINK** — total ?; splits: test (-1)
- **QBench** — total ?; splits: test (-1)
- **MuirBench** — total ?; splits: test (-1)
- **Mantis-Eval** — total ?; splits: test (-1)

## Metrics

- `benchmark score` **(primary)** — range: percent
  - Standard accuracy or benchmark-specific scoring metric (e.g., exact match, LLM-as-judge) applied per dataset. Values are reported as percentages.
- `average score` **(primary)** — range: percent
  - Arithmetic mean of the individual benchmark scores across all listed single- and multi-image benchmarks.

## Input / output format

**Input**: Single image or interleaved multi-image(s) paired with text prompts/questions.

**Output**: Text response (answer to the prompt/question).

## Scoring recipe

```python
scores = []
for dataset in BENCHMARKS:
    pred = model.generate(image, prompt)
    score = dataset.evaluate(pred, gold)  # Standard per-dataset metric
    scores.append(score)
average_score = sum(scores) / len(scores)
return average_score
```

## Common pitfalls

- Mixing up single-image (SI) and multi-image (MI) model variants when reporting scores, as the paper explicitly compares them.
- Using third-party leaderboard scores instead of official evaluation code, which the paper notes is necessary for fair comparison.
- Assuming a uniform scoring formula across all benchmarks, as each uses its own standard metric (e.g., MMVet uses LLM-as-judge, while others use exact match).

## Evidence (verbatim from paper)

> We evaluate our models (4B and 14B) on a comprehensive suite of multimodal benchmarks, assessing the model’s ability from multiple perspectives. Our evaluation covers general VQA benchmarks, domain knowledge, OCR ability, and hallucination. For models fine-tuned on interleaved multi-image datasets, we also evaluate their performance on common multi-image benchmarks. ... We report individual benchmark scores as well as the overall average score across all benchmarks, following the standard practice.

## Citation

```bibtex
@misc{xue2024xgenmm,
  title={xGen-MM (BLIP-3): A Family of Open Large Multimodal Models},
  author={Le Xue et al. (2024)},
  year={2024},
  note={arXiv:2408.08872}
}
```

- arXiv: 2408.08872