Gar Bench Eval

---
name: gar-bench-eval
description: Evaluates a multimodal LLM's ability to perform precise, context-aware visual understanding at the region level. It probes fine-grained perception, compositional reasoning across multiple visual prompts, and detailed localized captioning for both images and videos. Use when the user wants to benchmark on GAR-Bench-VQA, GAR-Bench-Cap, DLC-Bench, Ferret-Bench, MDVP-Bench, LVIS, PACO, VideoRefer-Bench, or asks about evaluating this task. Reports Overall score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.18876
  bibtex_key: wang2025graspanyregion
  confidence: high
---

# gar-bench-eval

> Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs — Wang et al. (2025) (arXiv:2510.18876, 2025)

## What this evaluates

Evaluates a multimodal LLM's ability to perform precise, context-aware visual understanding at the region level. It probes fine-grained perception, compositional reasoning across multiple visual prompts, and detailed localized captioning for both images and videos.

## Datasets

- **GAR-Bench-VQA** — total ?; splits: test (-1)
- **GAR-Bench-Cap** — total 204; splits: test (204)
- **DLC-Bench** — total ?; splits: test (-1)
- **Ferret-Bench** — total ?; splits: test (-1)
- **MDVP-Bench** — total ?; splits: test (-1)
- **LVIS** — total ?; splits: test (-1)
- **PACO** — total ?; splits: test (-1)
- **VideoRefer-Bench** — total ?; splits: test (-1)

## Metrics

- `Overall score` **(primary)** — range: percent
  - Percentage accuracy or LLM-judge score aggregated across subtasks (Perception, Reasoning, Color, Texture). For captioning tasks, GPT-4o is used as an automated judge to score generated descriptions against ground truth.
- `Semantic Similarity & IoU` — range: percent
  - Semantic similarity (Sim.) measures textual overlap/embedding similarity between predicted and ground-truth categories. IoU measures the intersection-over-union between predicted and ground-truth bounding boxes.

## Input / output format

**Input**: An image (or sampled video frames) with a specified region prompt (bounding box or mask), accompanied by a question or captioning instruction.

**Output**: Text response containing either a direct answer to the VQA question or a detailed localized caption describing the specified region.

## Scoring recipe

```python
# For VQA/Captioning
if judge == "GPT-4o":
    score = gpt4o_judge(model_output, ground_truth, cropped_image, mask)
else:
    score = exact_match(model_output, ground_truth)
overall = mean(scores)

# For LVIS/PACO
sim = semantic_similarity(predicted_cat, ground_truth_cat)
iou = compute_iou(predicted_box, ground_truth_box)
```

## Common pitfalls

- Captioning scores heavily depend on the LLM judge (GPT-4o) configuration; using cropped images with masks significantly changes scores compared to text-only judging.
- Video evaluation uses uniform 16-frame sampling and disables AnyRes for multi-frame tasks, which may underperform temporally dense videos.
- Zero-shot video transfer shows low scores on temporal/future prediction tasks because models are trained only on images.

## Evidence (verbatim from paper)

> Table 5: Results of category-level image recognition on LVIS [[15]] and PACO [[36]] following Osprey [[60]]. Our GAR-8B demonstrates a significant leap in performance, establishing a new state-of-the-art. It consistently outperforms all prior methods across every metric, achieving top scores of 93.6 semantic similarity and 88.7 semantic IoU on LVIS *[[15]], and 95.5 semantic similarity and 91.8 semantic IoU on PACO *[[36]].

## Citation

```bibtex
@misc{wang2025graspanyregion,
  title={Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.18876}
}
```

- arXiv: 2510.18876