Eagle2 Vlm Eval

---
name: eagle2-vlm-eval
description: Evaluates vision-language models on document understanding, chart and table reasoning, OCR, diagram comprehension, and general visual question answering. The protocol measures accuracy across a diverse suite of 14 established multimodal benchmarks to assess overall multimodal capability and robustness. Use when the user wants to benchmark on DocVQA, ChartQA, MMMU, MMB1.1, MathVista, or asks about evaluating this task. Reports OpenCompass.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.14818
  bibtex_key: li2025eagle2
  confidence: high
---

# eagle2-vlm-eval

> Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models — Li et al. (2025) (arXiv:2501.14818, 2025)

## What this evaluates

Evaluates vision-language models on document understanding, chart and table reasoning, OCR, diagram comprehension, and general visual question answering. The protocol measures accuracy across a diverse suite of 14 established multimodal benchmarks to assess overall multimodal capability and robustness.

## Datasets

- **DocVQA** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: val (-1)
- **MMB1.1** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test-mini (-1)

## Metrics

- `OpenCompass` **(primary)** — range: percent
  - Arithmetic mean of scores across OCRBench, MMStar, AI2D, MMMU, MMB1.1, MMVet, HallusionBench, and MathVista.
- `Benchmark Accuracy` — range: percent
  - Per-benchmark accuracy score. Computed using each dataset's official evaluation script (typically exact match for VQA or LLM-as-a-judge for open-ended tasks).

## Input / output format

**Input**: A single image paired with a natural language question or instruction prompt.

**Output**: A natural language text string containing the model's predicted answer.

## Scoring recipe

```python
def evaluate_model(model, dataset_splits):
    all_scores = []
    for split in dataset_splits:
        preds = model.generate(split.images, split.prompts)
        score = official_scorer(split.golds, preds)
        all_scores.append(score)
    opencompass_score = sum(all_scores) / len(all_scores)
    return opencompass_score, all_scores
```

## Common pitfalls

- Benchmarks use different evaluation splits (Test, Val, Test-Mini); the evaluation script must strictly match the split reported in the paper.
- OpenCompass is a custom composite metric averaging 8 specific benchmarks, not a standard public leaderboard score.
- Some models have missing scores on certain benchmarks (e.g., InfoVQA, RWQA), which must be handled as NaN rather than zero when computing averages.

## Evidence (verbatim from paper)

> *: We obtain the OpenCompass score by averaging across Eagle benchmarks (OCRBench, MMStar, AI2D, MMMU, MMB1.1, MMVet, HallusionBench, and MathVista).

## Citation

```bibtex
@misc{li2025eagle2,
  title={Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2501.14818}
}
```

- arXiv: 2501.14818