Vision R1 Eval

---
name: vision-r1-eval
description: Evaluates Large Vision-Language Models on their ability to detect, localize, and ground objects in images across diverse and challenging scenarios, including in-domain dense detection, out-of-domain real-world settings, and generalization to unseen categories or scenes. Use when the user wants to benchmark on MSCOCO Val2017, ODINW-13, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.18013
  bibtex_key: zhan2025visionr1
  confidence: high
---

# vision-r1-eval

> Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning — Zhan et al. (2025) (arXiv:2503.18013, 2025)

## What this evaluates

Evaluates Large Vision-Language Models on their ability to detect, localize, and ground objects in images across diverse and challenging scenarios, including in-domain dense detection, out-of-domain real-world settings, and generalization to unseen categories or scenes.

## Datasets

- **MSCOCO Val2017** — total ?; splits: val (-1)
- **ODINW-13** — total ?; splits: test (-1)

## Metrics

- `mAP` **(primary)** — range: [0, 100]
  - Mean Average Precision across all object categories, computed by averaging the Area Under the Precision-Recall curve at IoU thresholds of 0.50 to 0.95 (standard COCO protocol).
- `AP50` — range: [0, 100]
  - Average Precision at Intersection over Union (IoU) threshold of 0.50.
- `AP75` — range: [0, 100]
  - Average Precision at Intersection over Union (IoU) threshold of 0.75.
- `AR100` — range: [0, 100]
  - Average Recall at a maximum of 100 detected objects per image.

## Input / output format

**Input**: RGB image paired with a text prompt specifying the localization task (e.g., object detection query, visual grounding instruction, or referring expression).

**Output**: A list of predicted bounding boxes (coordinates) and deterministic category labels for each detected object.

## Scoring recipe

```python
def compute_coco_metrics(predictions, ground_truths):
    # 1. Group predictions and ground truths by category
    # 2. For each category, compute IoU between predicted and GT boxes
    # 3. Sort predictions by deterministic match and compute PR curve
    # 4. Calculate AP at IoU thresholds 0.50, 0.75, and 0.50:0.95
    # 5. Average AP across all categories to get mAP
    # 6. Compute AR@100 by counting true positives in top-100 predictions per image
    return mAP, AP50, AP75, AR100
```

## Common pitfalls

- ODINW evaluation follows a visual grounding setting rather than standard object detection, requiring careful prompt formatting.
- Out-of-domain evaluation relaxes the strict 'unseen category + unseen scene' constraint; only one of the two needs to be absent during post-training.
- LVLMs output deterministic category labels instead of class probabilities, so box matching relies primarily on spatial accuracy rather than Hungarian matching with confidence scores.

## Evidence (verbatim from paper)

> The results in Tab. [1] demonstrate the broad effectiveness of the Vision-R1 in object localization tasks. When applied to the Griffon-G model, which excels in object detection, Vision-R1 further improves its performance by 1.8 on COCO and achieves an average mAP increase of 2.5 on ODINW-13.

## Citation

```bibtex
@misc{zhan2025visionr1,
  title={Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning},
  author={Zhan et al. (2025)},
  year={2025},
  note={arXiv:2503.18013}
}
```

- arXiv: 2503.18013