Groundingme Eval

---
name: groundingme-eval
description: Evaluates multimodal large language models' (MLLMs) visual grounding capabilities across four dimensions: discriminative object distinction, spatial relational understanding, handling occlusion/size constraints, and the ability to reject ungroundable queries. It measures how well models can localize objects in images and whether they hallucinate or correctly refuse impossible requests. Use when the user wants to benchmark on GroundingME, or asks about evaluating this task. Reports Accuracy@0.5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.17495
  bibtex_key: [REDACTED]
  confidence: high
---

# groundingme-eval

> GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation — Li et al. (2025) (arXiv:2512.17495, 2025)

## What this evaluates

Evaluates multimodal large language models' (MLLMs) visual grounding capabilities across four dimensions: discriminative object distinction, spatial relational understanding, handling occlusion/size constraints, and the ability to reject ungroundable queries. It measures how well models can localize objects in images and whether they hallucinate or correctly refuse impossible requests.

## Datasets

- **GroundingME** — total ?; splits: test (-1)

## Metrics

- `Accuracy@0.5` **(primary)** — range: percent
  - The percentage of test samples where the Intersection over Union (IoU) between the predicted bounding box and the ground-truth bounding box exceeds 0.5.

## Input / output format

**Input**: An input image paired with a textual description or query, formatted using a unified prompt template that specifies viewpoint references, allowed output counts, and strict formatting constraints.

**Output**: Bounding box coordinates for the target object(s), strictly following the template's format constraints and output count limits.

## Scoring recipe

```python
def accuracy_at_0_5(predictions, ground_truths):
    correct = 0
    for pred, gt in zip(predictions, ground_truths):
        if calculate_iou(pred, gt) > 0.5:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Models often hallucinate objects for invalid queries instead of correctly rejecting them, severely depressing rejection scores.
- Reporting only the overall average accuracy masks significant performance stratification across subcategories (e.g., high discriminative scores vs. near-zero rejection scores).
- The default evaluation protocol uses greedy decoding (temperature=0), which may not reflect each model's peak performance.

## Evidence (verbatim from paper)

> For the evaluation metric, we adopt the widely-used Accuracy@0.5, which represents the proportion of total samples where the Intersection over Union (IoU) between the ground-truth and predicted bounding box exceeds 0.5.

## Citation

```bibtex
@misc{li2025groundingme,
  title={GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.17495}
}
```

- arXiv: 2512.17495