V Triune Eval

---
name: v-triune-eval
description: Evaluates vision-language models on a unified suite of visual reasoning and perception tasks, measuring generalization across real-world benchmarks, mathematical reasoning, and object detection/grounding capabilities. Use when the user wants to benchmark on MEGA-Bench Core, MMMU, MathVista, COCO, OVDEval, CountBench, OCRBench, ScreenSpot-Pro, or asks about evaluating this task. Reports MEGA-Bench Core weighted average.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.18129
  bibtex_key: ma2025one
  confidence: high
---

# v-triune-eval

> One RL to See Them All: Visual Triple Unified Reinforcement Learning — Yan Ma et al. (2025) (arXiv:2505.18129, 2025)

## What this evaluates

Evaluates vision-language models on a unified suite of visual reasoning and perception tasks, measuring generalization across real-world benchmarks, mathematical reasoning, and object detection/grounding capabilities.

## Datasets

- **MEGA-Bench Core** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: val (-1)
- **MathVista** — total ?; splits: testmini (-1)
- **COCO** — total ?; splits: val-2017 (-1)
- **OVDEval** — total ?; splits: test (-1)
- **CountBench** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: v2 (-1)
- **ScreenSpot-Pro** — total ?; splits: test (-1)

## Metrics

- `MEGA-Bench Core weighted average` **(primary)** — range: percent
  - Weighted average score across 440 diverse tasks in the MEGA-Bench Core subset, computed using the official evaluation implementation.
- `mAP / mAP@50` — range: percent
  - Mean Average Precision and mAP at IoU threshold 0.5. Computed as the mean of sample-level metrics rather than over the entire dataset due to the absence of confidence scores in model outputs.
- `NMS-AP` — range: percent
  - Task-wide mean Average Precision after Non-Maximum Suppression, computed as the mean of sample-level metrics.
- `Accuracy` — range: percent
  - Exact match or rule-based correctness for reasoning and OCR tasks. For MMMU, both rule-based and GPT-4o-based scoring are reported.

## Input / output format

**Input**: Image(s) paired with task-specific text prompts. For detection and grounding tasks, prompts instruct the model to identify instances of a given object and format the output in a specific reasoning-answer format.

**Output**: Textual reasoning and answers. For perception tasks, bounding box coordinates represented as values relative to the original input image dimensions.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type == 'MEGA-Bench':
        return weighted_average_score(predictions, golds)
    elif task_type in ['COCO', 'OVDEval']:
        sample_scores = [compute_sample_iou(pred, gold) for pred, gold in zip(predictions, golds)]
        return mean(sample_scores)
    elif task_type in ['MMMU', 'MathVista', 'CountBench', 'OCRBench', 'ScreenSpot-Pro']:
        return accuracy(predictions, golds)
    return None
```

## Common pitfalls

- mAP and NMS-AP are computed as the mean of sample-level metrics rather than over the entire dataset due to missing confidence scores.
- MMMU uses both rule-based and GPT-4o scoring, which may introduce variability in results.
- Greedy decoding is used for evaluation, while temperature=1.0 is used during training rollout.

## Evidence (verbatim from paper)

> We report both IoU and mAP for COCO and OVDEval. Notably, mAP and NMS-mAP are computed as the mean of sample-level metrics, rather than over the entire dataset, due to the absence of confidence scores in model outputs.

## Citation

```bibtex
@misc{ma2025one,
  title={One RL to See Them All: Visual Triple Unified Reinforcement Learning},
  author={Yan Ma et al. (2025)},
  year={2025},
  note={arXiv:2505.18129}
}
```

- arXiv: 2505.18129