Mme Realworld Vbench Eval

---
name: mme-realworld-vbench-eval
description: This evaluation probes a model's ability to perform high-resolution visual reasoning and fine-grained grounding on complex, real-world images. It specifically tests whether the model can accurately localize relevant visual regions and correctly answer multiple-choice questions without explicit grounding supervision. Use when the user wants to benchmark on MME-Realworld, V* Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.05920
  bibtex_key: huang2025mgpo
  confidence: high
---

# mme-realworld-vbench-eval

> High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning — Xinyu Huang et al. (2025) (arXiv:2507.05920, 2025)

## What this evaluates

This evaluation probes a model's ability to perform high-resolution visual reasoning and fine-grained grounding on complex, real-world images. It specifically tests whether the model can accurately localize relevant visual regions and correctly answer multiple-choice questions without explicit grounding supervision.

## Datasets

- **MME-Realworld** — total 23609; splits: train (21690), test (1919)
- **V* Bench** — total 191; splits: test (191)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of samples) * 100.

## Input / output format

**Input**: A high-resolution image paired with a multiple-choice question (options A–E).

**Output**: The model must output the selected answer letter (A, B, C, D, or E) enclosed in \boxed{}, optionally preceded by grounding coordinates in JSON format.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        answer = extract_answer_from_box(pred)
        if answer == gold:
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Images exceeding 1,003,520 pixels are resized to fit the model's token limit, which can degrade performance on tasks requiring fine-grained details if not handled carefully.
- The evaluation strictly requires the answer letter to be placed within \boxed{} for reliable parsing; deviations cause accuracy calculation failures.
- V* Bench is an out-of-distribution (OOD) benchmark while MME-Realworld is in-distribution (ID); results must be reported separately to avoid conflating generalization with memorization.

## Evidence (verbatim from paper)

> All datasets employ the multiple-choice question format, and model performance is consistently measured by accuracy on both the in-distribution (MME-Realworld) and out-of-distribution (V* Bench) test sets.

## Citation

```bibtex
@misc{huang2025mgpo,
  title={High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning},
  author={Xinyu Huang et al. (2025)},
  year={2025},
  note={arXiv:2507.05920}
}
```

- arXiv: 2507.05920