Grounder Eval

---
name: grounder-eval
description: This benchmark evaluates a model's ability to localize arbitrary natural language phrases within images. It probes phrase grounding capabilities by requiring the model to attend to relevant image regions and select a bounding box that matches the textual description, without relying on explicit bounding box supervision during training. Use when the user wants to benchmark on Flickr 30k Entities, ReferItGame, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1511.03745
  bibtex_key: [REDACTED]
  confidence: high
---

# grounder-eval

> Grounding of Textual Phrases in Images by Reconstruction — Rohrbach et al. (2015) (arXiv:1511.03745, 2015)

## What this evaluates

This benchmark evaluates a model's ability to localize arbitrary natural language phrases within images. It probes phrase grounding capabilities by requiring the model to attend to relevant image regions and select a bounding box that matches the textual description, without relying on explicit bounding box supervision during training.

## Datasets

- **Flickr 30k Entities** — total 31000; splits: train (29783), val (1000), test (1000)
- **ReferItGame** — total 20000; splits: train (9000), val (1000), test (10000)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Ratio of phrases for which the attended bounding box overlaps with the ground-truth box by more than 0.5 IOU. For phrases corresponding to multiple boxes, the union of boxes is used as ground truth.

## Input / output format

**Input**: An image and a natural language phrase describing a target region within that image.

**Output**: A single bounding box selected from a set of pre-computed proposals (or attention weights over those proposals).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_boxes):
    correct = 0
    for pred_box, gold_box in zip(predictions, gold_boxes):
        iou = intersection_over_union(pred_box, gold_box)
        if iou > 0.5:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Multiple boxes per phrase: the union of all corresponding ground-truth boxes is used as the single target.
- Proposal generation differs by dataset: Selective Search is used for Flickr 30k Entities, while Edge Boxes are used for ReferItGame.
- Semi-supervised training only uses phrase-box pairs where a proposal overlaps the ground truth by >0.5 IOU.
- Visual features vary: VGG-CLS (ImageNet), VGG-DET (PASCAL fine-tuned), and VGG+SPAT (ReferItGame specific spatial features).

## Evidence (verbatim from paper)

> At test time we compute the accuracy as the ratio of phrases for which the attended box overlaps with the ground-truth box by more than 0.5 IOU. For consistency with [[35]], in such cases we consider the union of the boxes as ground truth.

## Citation

```bibtex
@misc{rohrbach2015grounding,
  title={Grounding of Textual Phrases in Images by Reconstruction},
  author={Rohrbach et al. (2015)},
  year={2015},
  note={arXiv:1511.03745}
}
```

- arXiv: 1511.03745