Finevision Eval

---
name: finevision-eval
description: Evaluates vision-language models on a diverse suite of 11 multimodal benchmarks covering visual question answering, chart understanding, document parsing, and general multimodal reasoning. Additionally probes GUI/agentic capabilities on screen interaction tasks. Use when the user wants to benchmark on AI2D, ChartQA, DocVQA, InfoVQA, MME, MMMU, ScienceQA, MMStar, OCRBench, TextVQA, SEED-Bench, Screenspot-V2, Screenspot-Pro, or asks about evaluating this task. Reports mean normalized performance (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.17269
  bibtex_key: [REDACTED]
  confidence: high
---

# finevision-eval

> FineVision: Open Data Is All You Need — Wiedmann et al. (2025) (arXiv:2510.17269, 2025)

## What this evaluates

Evaluates vision-language models on a diverse suite of 11 multimodal benchmarks covering visual question answering, chart understanding, document parsing, and general multimodal reasoning. Additionally probes GUI/agentic capabilities on screen interaction tasks.

## Datasets

- **AI2D** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **InfoVQA** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **ScienceQA** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **SEED-Bench** — total ?; splits: test (-1)
- **Screenspot-V2** — total ?; splits: test (-1)
- **Screenspot-Pro** — total ?; splits: test (-1)

## Metrics

- `mean normalized performance (%)` **(primary)** — range: [0, 100]
  - Each benchmark's raw score is min-max normalized to [0,100]. The final metric is the arithmetic mean of these normalized scores across all 11 benchmarks.

## Input / output format

**Input**: Image(s) and text prompt/question per instance, formatted according to each benchmark's specification.

**Output**: Text response generated by the model, evaluated against ground truth using benchmark-specific evaluators.

## Scoring recipe

```python
scores = []
for bench in benchmarks:
    raw = evaluate(model, bench)
    norm = min_max_normalize(raw, bench.min, bench.max)
    scores.append(norm)
final_metric = sum(scores) / len(scores)
```

## Common pitfalls

- Benchmark scores are min-max normalized to [0,100] before averaging, which can obscure task-specific performance differences.
- Test data contamination varies across datasets (1.02% for FineVision vs 2.15–3.05% for baselines), directly impacting reported scores.
- Small models often fail completely at the base stage, requiring fine-tuning to show meaningful results on GUI benchmarks.

## Evidence (verbatim from paper)

> mean normalized performance (%) across 11 evaluation benchmarks (higher is better), with the training step shown in thousands ($\times 10^{3}$). Each benchmark score is min–max normalized to [0,100] and averaged per evaluation step; the model trained on FineVision (blue) leads throughout the second half of training and attains the best final score.

## Citation

```bibtex
@misc{wiedmann2025finevision,
  title={FineVision: Open Data Is All You Need},
  author={Wiedmann et al. (2025)},
  year={2025},
  note={arXiv:2510.17269}
}
```

- arXiv: 2510.17269