Refexp Eval

---
name: refexp-eval
description: Evaluates a model's ability to generate unambiguous, context-aware text descriptions for specific objects in an image, and to comprehend those descriptions by correctly localizing the target object via bounding box prediction. Use when the user wants to benchmark on G-Ref, UNC-Ref, or asks about evaluating this task. Reports precision@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1511.02283
  bibtex_key: [REDACTED]
  confidence: high
---

# refexp-eval

> Generation and Comprehension of Unambiguous Object Descriptions — Mao et al. (2015) (arXiv:1511.02283, 2015)

## What this evaluates

Evaluates a model's ability to generate unambiguous, context-aware text descriptions for specific objects in an image, and to comprehend those descriptions by correctly localizing the target object via bounding box prediction.

## Datasets

- **G-Ref** — total 54822; splits: train (44822), val (5000), test (5000); repo https://github.com/mjhuela/Google_Refexp_toolbox
- **UNC-Ref** — total 50000; splits: train (40000), val (5000), test (5000); repo https://github.com/mjhuela/Google_Refexp_toolbox

## Metrics

- `precision@1` **(primary)** — range: [0, 1]
  - For each image, compute IoU between the predicted bounding box and the ground truth box. If IoU > 0.5, score is 1; else 0. Average across all images.
- `human_eval_score` — range: percent
  - Percentage of automatically generated descriptions judged by AMT workers as better or equal to human-generated captions for the same object.

## Input / output format

**Input**: Image + text description (comprehension task) or Image + target object bounding box (generation task).

**Output**: Predicted bounding box coordinates (comprehension) or a natural language description string (generation).

## Scoring recipe

```python
def compute_precision_at_1(pred_boxes, gt_boxes):
    ious = [iou(p, g) for p, g in zip(pred_boxes, gt_boxes)]
    scores = [1.0 if iou > 0.5 else 0.0 for iou in ious]
    return sum(scores) / len(scores)
```

## Common pitfalls

- Standard captioning metrics (BLEU, CIDEr) are unreliable for this task as they do not measure semantic uniqueness.
- End-to-end scores can be artificially inflated because generation and comprehension models share parameters, allowing the system to 'communicate' in its own idiosyncratic language.
- Human evaluation requires blinding workers to machine vs. human origin to prevent bias.

## Evidence (verbatim from paper)

> The comprehension task is easy to evaluate: we simply compute the Intersection over Union (IoU) ratio between the true and predicted bounding box. If IoU exceeds 0.5, we call the detection a true positive, otherwise it is a false positive (this is equivalent to computing the precision@1 measure). We then average this score over all images.

## Citation

```bibtex
@misc{mao2015generation,
  title={Generation and Comprehension of Unambiguous Object Descriptions},
  author={Mao et al. (2015)},
  year={2015},
  note={arXiv:1511.02283}
}
```

- arXiv: 1511.02283