Referring Expression Comprehension Eval

---
name: referring-expression-comprehension-eval
description: Probes a model's ability to ground referring expressions in images by understanding spatial and relational context between objects. It evaluates weakly supervised region proposal scoring and context pooling mechanisms without requiring explicit bounding box annotations for context regions. Use when the user wants to benchmark on Google RefExp, UNC RefExp, or asks about evaluating this task. Reports Precision@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1608.00525
  bibtex_key: nagaraja2016modeling
  confidence: high
---

# referring-expression-comprehension-eval

> Modeling Context Between Objects for Referring Expression Understanding — Nagaraja et al. (2016) (arXiv:1608.00525, 2016)

## What this evaluates

Probes a model's ability to ground referring expressions in images by understanding spatial and relational context between objects. It evaluates weakly supervised region proposal scoring and context pooling mechanisms without requiring explicit bounding box annotations for context regions.

## Datasets

- **Google RefExp** — total ?; splits: train (85408), val (9602)
- **UNC RefExp** — total ?; splits: train (120624), val (10834), testA (5657), testB (5095)

## Metrics

- `Precision@1` **(primary)** — range: percent
  - Averaged over all referring expressions: (1/N) * Σ [IoU(pred_box, gt_box) > 0.5], where a prediction is a true positive if the Intersection over Union exceeds 0.5.

## Input / output format

**Input**: An image, a referring expression (text), and a set of region proposals (bounding boxes) with extracted CNN features.

**Output**: A single predicted bounding box (x_min, y_min, x_max, y_max) corresponding to the referred object in the image.

## Scoring recipe

```python
def compute_precision_at_1(predictions, ground_truths):
    correct = 0
    for pred_box, gt_box in zip(predictions, ground_truths):
        iou = intersection_over_union(pred_box, gt_box)
        if iou > 0.5:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- The original Google RefExp partition had overlapping images across train/val/test splits, causing data leakage; the authors explicitly created a non-overlapping partition to fix this.
- UNC RefExp TestA (person-centric) and TestB (object-centric) splits differ significantly in expression terseness, causing context modeling to help less on TestA.
- Testing uses noisy-or pooling over up to 10 sampled context regions, making scores sensitive to the quality of region proposals (MCG vs. ground truth).

## Evidence (verbatim from paper)

> The evaluation is performed by measuring the Intersection over Union (IoU) ratio between a groundtruth box and the top predicted box for a referring expression. If the IoU >0.5, the prediction is considered a true positive and this is the Precision@1 score. The scores are then averaged over all referring expressions.

## Citation

```bibtex
@misc{nagaraja2016modeling,
  title={Modeling Context Between Objects for Referring Expression Understanding},
  author={Nagaraja et al. (2016)},
  year={2016},
  note={arXiv:1608.00525}
}
```

- arXiv: 1608.00525