Openvlthinkerv2 Eval

---
name: openvlthinkerv2-eval
description: Evaluates a multimodal reasoning model's capability across diverse visual tasks, including general and mathematical VQA, document understanding, spatial reasoning, and visual grounding. The protocol tests the model's ability to balance fine-grained perception with multi-step reasoning under a unified RL training framework. Use when the user wants to benchmark on MMMU, MMBench, MMStar, ChartQA, DocVQA, OCRBench, InfoVQA, EmbSpatial, RefSpatial, RoboSpatial, RefCOCO, RefCOCO+, RefCOCOg, or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.08539
  bibtex_key: [REDACTED]
  confidence: high
---

# openvlthinkerv2-eval

> OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks — Wenbo Hu et al. (arXiv:2604.08539, 2026)

## What this evaluates

Evaluates a multimodal reasoning model's capability across diverse visual tasks, including general and mathematical VQA, document understanding, spatial reasoning, and visual grounding. The protocol tests the model's ability to balance fine-grained perception with multi-step reasoning under a unified RL training framework.

## Datasets

- **MMMU** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **ChartQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **InfoVQA** — total ?; splits: test (-1)
- **EmbSpatial** — total ?; splits: test (-1)
- **RefSpatial** — total ?; splits: test (-1)
- **RoboSpatial** — total ?; splits: test (-1)
- **RefCOCO** — total ?; splits: val (-1)
- **RefCOCO+** — total ?; splits: val (-1)
- **RefCOCOg** — total ?; splits: val (-1)

## Metrics

- `score` **(primary)** — range: percent
  - Standard accuracy or composite benchmark score reported as a percentage or raw value depending on the specific benchmark. Calculated as the proportion of correctly predicted answers or bounding boxes matching ground truth within a specified threshold.

## Input / output format

**Input**: Multimodal inputs consisting of an image and a text prompt/question.

**Output**: Text response containing the answer, reasoning chain, or bounding box coordinates.

## Scoring recipe

```python
def calculate_score(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if is_correct(pred, gold_item):
            correct += 1
    return (correct / len(gold)) * 100

def is_correct(pred, gold):
    if gold_type == 'bbox':
        return compute_iou(pred, gold) >= 0.5
    return pred.strip().lower() == gold.strip().lower()
```

## Common pitfalls

- Models must be evaluated using identical generation hyperparameters to baselines to ensure fair comparison.
- Reward shaping and data filtering during RL training can cause overfitting to specific task distributions, inflating scores on in-domain benchmarks while degrading out-of-distribution generalization.
- Composite benchmarks like OCRBench aggregate multiple sub-tasks, making it difficult to isolate specific capability improvements.

## Evidence (verbatim from paper)

> Table 3: Evaluation results for Grounding task. We report scores on val splits. Our model consistently outperforms previous baselines.

## Citation

```bibtex
@misc{hu2026openvlthinkerv2,
  title={OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks},
  author={Wenbo Hu et al.},
  year={2026},
  note={arXiv:2604.08539}
}
```

- arXiv: 2604.08539