Llava Onevision 1.5 Eval

---
name: llava-onevision-1.5-eval
description: This evaluation probes the multimodal reasoning, visual question answering, OCR, and chart understanding capabilities of large multimodal models. It tests the model's ability to process high-resolution images, extract fine-grained text, and perform complex reasoning across diverse visual domains. Use when the user wants to benchmark on MMStar, MMEBench, MME-RealWorld, SeedBench, CV-Bench, RealWorldQA, MathVista, WeMath, MathVision, MMMU, MMMU-Pro, ChartQA, CharXiv, DocVQA, OCRBench, AI2D, InfoVQA, PixmoCount, CountBench, VL-RewardBench, V*, or asks about evaluating this task. Reports accuracy / benchmark-specific score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.23661
  bibtex_key: an2025llavaonevision15
  confidence: high
---

# llava-onevision-1.5-eval

> LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training — Xiang An et al. (arXiv:2509.23661, 2025)

## What this evaluates

This evaluation probes the multimodal reasoning, visual question answering, OCR, and chart understanding capabilities of large multimodal models. It tests the model's ability to process high-resolution images, extract fine-grained text, and perform complex reasoning across diverse visual domains.

## Datasets

- **MMStar** — total ?; splits: test (-1)
- **MMEBench** — total ?; splits: test (-1)
- **MME-RealWorld** — total ?; splits: test (-1)
- **SeedBench** — total ?; splits: test (-1)
- **CV-Bench** — total ?; splits: test (-1)
- **RealWorldQA** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **WeMath** — total ?; splits: test (-1)
- **MathVision** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: val (-1)
- **MMMU-Pro** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **CharXiv** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **AI2D** — total ?; splits: test (-1)
- **InfoVQA** — total ?; splits: test (-1)
- **PixmoCount** — total ?; splits: test (-1)
- **CountBench** — total ?; splits: test (-1)
- **VL-RewardBench** — total ?; splits: test (-1)
- **V*** — total ?; splits: test (-1)

## Metrics

- `accuracy / benchmark-specific score` **(primary)** — range: percent
  - Standard exact-match or normalized accuracy computed per benchmark, aggregated using the LMMs-Eval framework. Scores are reported as percentages.

## Input / output format

**Input**: Image and text prompt (question or instruction) provided to the model.

**Output**: Text response (answer string).

## Scoring recipe

```python
def compute_metric(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_clean = normalize_whitespace(pred)
        gold_clean = normalize_whitespace(gold)
        if pred_clean == gold_clean or matches_regex(pred_clean, gold_clean):
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Using non-default prompts or temperature settings can drastically alter scores on benchmarks like MME or MathVista.
- OCR benchmarks are highly sensitive to image resolution and tiling strategies; inconsistent preprocessing leads to unfair comparisons.

## Evidence (verbatim from paper)

> We use LMMs-Eval Zhang et al. (2024) with the default prompt to evaluate the performance of LLaVA-OneVision-1.5 across multiple benchmarks in four categories of downstream tasks:(1) General Visual Question Answering (VQA): MMStar (Chen et al., 2024a), MMEBench series (Fu et al., 2023), MME-RealWorld series (Zhang et al., 2025), SeedBench (Li et al., 2024b), Seed-Bench-2-Plus (Li et al., 2024a), CV-Bench (Tong et al., 2024), and RealWorldQA (Corp., 2024). (2) Multimodal Reasoning: MathVista (Lu et al., 2024), WeMath (Qiao et al., 2025), MathVision (Wang et al., 2024a), MMMU (Yue et al., 2024), and MMMU-Pro series (Yue et al., 2025).

## Citation

```bibtex
@misc{an2025llavaonevision15,
  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
  author={Xiang An et al.},
  year={2025},
  note={arXiv:2509.23661}
}
```

- arXiv: 2509.23661