Innovator Vl Eval

---
name: innovator-vl-eval
description: Evaluates multimodal large language models across general vision, mathematical reasoning, and specialized scientific domains to measure visual perception, instruction following, and domain-specific knowledge retention. Use when the user wants to benchmark on AI2D, OCRBench, ChartQA, MMMU(Val), MMMU-Pro (Standard), MMStar, VStar-Bench, MMBench-EN, MME-RealWorld, DocVQA(Val), InfoVQA(Val), SEED-Bench, SEED-Bench-2-plus, RealWorldQA, MathVision, MathVerse, MathVista, WeMath, ScienceQA, RxnBench, MolParse, OpenRxn, EMVista, SuperChem, SmolInstruct, ProteinLMBench, SFE, MicroVQA, MSEarth-MCQ, XLRS-Bench-lite, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.19325
  bibtex_key: wen2026innovatorvl
  confidence: high
---

# innovator-vl-eval

> Innovator-VL: A Multimodal Large Language Model for Scientific Discovery — Wen et al. (2026) (arXiv:2601.19325, 2026)

## What this evaluates

Evaluates multimodal large language models across general vision, mathematical reasoning, and specialized scientific domains to measure visual perception, instruction following, and domain-specific knowledge retention.

## Datasets

- **AI2D** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **MMMU(Val)** — total ?; splits: val (-1)
- **MMMU-Pro (Standard)** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **VStar-Bench** — total ?; splits: test (-1)
- **MMBench-EN** — total ?; splits: dev (-1), test (-1)
- **MME-RealWorld** — total ?; splits: test (-1)
- **DocVQA(Val)** — total ?; splits: val (-1)
- **InfoVQA(Val)** — total ?; splits: val (-1)
- **SEED-Bench** — total ?; splits: test (-1)
- **SEED-Bench-2-plus** — total ?; splits: test (-1)
- **RealWorldQA** — total ?; splits: test (-1)
- **MathVision** — total ?; splits: test (-1), mini (-1)
- **MathVerse** — total ?; splits: mini (-1)
- **MathVista** — total ?; splits: mini (-1)
- **WeMath** — total ?; splits: test (-1)
- **ScienceQA** — total ?; splits: test (-1)
- **RxnBench** — total ?; splits: en (-1), zh (-1)
- **MolParse** — total ?; splits: test (-1); HF `InnovatorLab/MolParse`
- **OpenRxn** — total ?; splits: test (-1); HF `InnovatorLab/OpenRxn`
- **EMVista** — total ?; splits: test (-1); HF `InnovatorLab/EMVista`
- **SuperChem** — total ?; splits: en (-1), zh (-1)
- **SmolInstruct** — total ?; splits: test (-1)
- **ProteinLMBench** — total ?; splits: test (-1)
- **SFE** — total ?; splits: en (-1), zh (-1)
- **MicroVQA** — total ?; splits: test (-1)
- **MSEarth-MCQ** — total ?; splits: test (-1)
- **XLRS-Bench-lite** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions across multiple-choice or open-ended prompts. Computed per benchmark and averaged within capability categories (general, math & reasoning, science).

## Input / output format

**Input**: Multimodal instances containing domain-specific or general images paired with text prompts/questions.

**Output**: Text responses generated by the model, typically multiple-choice selections or short-form answers, decoded deterministically (temperature 0.0, top-p 1.0).

## Scoring recipe

```python
predictions = []
for dataset in benchmarks:
    for instance in dataset.test:
        output = model.generate(image=instance.image, prompt=instance.question, temperature=0.0, top_p=1.0)
        predictions.append(extract_answer(output))
    acc = compute_accuracy(predictions, dataset.gold_answers)
    category_scores[dataset.category].append(acc)
final_score = average(category_scores)
```

## Common pitfalls

- Averaging accuracy across highly heterogeneous benchmarks (e.g., OCR vs. molecular parsing) can obscure domain-specific strengths or weaknesses.
- Using non-deterministic decoding (temperature > 0) on multiple-choice scientific benchmarks introduces significant score variance and reduces reproducibility.
- The lmms-eval framework requires specific prompt templates and answer extraction regexes per dataset; mismatched configurations yield artificially low scores.

## Evidence (verbatim from paper)

> We run all evaluations using the lmms-eval framework (Zhang et al., [2024b]). For Innovator-VL-8B-Instruct and Innovator-VL-8B-Thinking, we use deterministic decoding (temperature 0.0, top-p 1.0) across all benchmarks. ... Figure 5: Token efficiency comparison across vision reasoning benchmarks. ... (b) Accuracy-to-token ratio, which measures the reasoning efficiency, demonstrating that Innovator-VL-8B-Thinking achieves 1.4x to 2x higher accuracy-to-token ratio than MiMo-VL-7B-RL and 3.9x to 4.3x higher than Intern-S1-mini.

## Citation

```bibtex
@misc{wen2026innovatorvl,
  title={Innovator-VL: A Multimodal Large Language Model for Scientific Discovery},
  author={Wen et al. (2026)},
  year={2026},
  note={arXiv:2601.19325}
}
```

- arXiv: 2601.19325