Veroeval Eval

---
name: veroeval-eval
description: Evaluates general visual reasoning capabilities across a diverse set of 30 benchmarks spanning six task categories, including chart/OCR, STEM, spatial/action, knowledge/recognition, grounding, and captioning/instruction following. Use when the user wants to benchmark on VeroEval, or asks about evaluating this task. Reports overall averages.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.04917
  bibtex_key: sarch2026vero
  confidence: high
---

# veroeval-eval

> Vero: An Open RL Recipe for General Visual Reasoning — Sarch et al. (2026) (arXiv:2604.04917, 2026)

## What this evaluates

Evaluates general visual reasoning capabilities across a diverse set of 30 benchmarks spanning six task categories, including chart/OCR, STEM, spatial/action, knowledge/recognition, grounding, and captioning/instruction following.

## Datasets

- **VeroEval** — total ?; splits: test (-1); repo https://github.com/zlab-princeton/vero

## Metrics

- `overall averages` **(primary)** — range: percent
  - Mean of the official scores reported by each of the 30 constituent benchmarks. Scores are aggregated at the task-category level and overall.

## Input / output format

**Input**: Multimodal prompts (images + text) per benchmark, evaluated with one sampled decode per example.

**Output**: Model-generated text responses (answers, reasoning traces, or captions) depending on the benchmark.

## Scoring recipe

```python
scores = []
for bench in veroeval_benchmarks:
    preds = model.generate(prompts[bench], max_tokens=..., temperature=...)
    score = lmms_eval.evaluate(bench, preds, gold) # Official protocol per benchmark
    scores.append(score)
overall_averages = sum(scores) / len(scores)
```

## Common pitfalls

- Using different decoding parameters than specified per model family (qwen25vl vs qwen3vl).
- Not following each benchmark's official evaluation protocol, which varies by task type.
- LLM/VLM judge prompts and sampling parameters (Temp=0.7, TopP=0.8) can introduce variance if not fixed.

## Evidence (verbatim from paper)

> Our best models Vero-Qwen3T-8B and Vero-Qwen3I-8B achieve the highest overall averages (65.9 and 66.0, respectively) among all 8B-parameter VLMs evaluated, outperforming baselines across the six task categories. We evaluate all models using the lmms-eval framework, following the official evaluation protocols specified by each benchmark’s authors.

## Citation

```bibtex
@misc{sarch2026vero,
  title={Vero: An Open RL Recipe for General Visual Reasoning},
  author={Sarch et al. (2026)},
  year={2026},
  note={arXiv:2604.04917}
}
```

- arXiv: 2604.04917