Nl Object Retrieval Eval

---
name: nl-object-retrieval-eval
description: Evaluates a model's ability to ground natural language queries to specific regions within images by scoring candidate bounding boxes. It probes spatial reasoning, contextual understanding, and cross-modal alignment between text and visual features. Use when the user wants to benchmark on ReferIt, Kitchen, or asks about evaluating this task. Reports P@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1511.04164
  bibtex_key: hu2015natural
  confidence: high
---

# nl-object-retrieval-eval

> Natural Language Object Retrieval — Hu et al. (2015) (arXiv:1511.04164, 2015)

## What this evaluates

Evaluates a model's ability to ground natural language queries to specific regions within images by scoring candidate bounding boxes. It probes spatial reasoning, contextual understanding, and cross-modal alignment between text and visual features.

## Datasets

- **ReferIt** — total 20000; splits: trainval (10000), test (10000)
- **Kitchen** — total 606; splits: trainval (300), test (306)

## Metrics

- `P@1` **(primary)** — range: percent
  - Top-1 precision: the percentage of queries where the highest-scoring candidate region overlaps with the ground truth bounding box by at least 50% IoU.
- `R@1` — range: percent
  - Recall@1: the percentage of queries where the highest-scoring candidate region overlaps with the ground truth by at least 50% IoU.
- `R@10` — range: percent
  - Recall@10: the percentage of queries where at least one of the top-10 highest-scoring candidates overlaps with the ground truth by at least 50% IoU.

## Input / output format

**Input**: An image, a set of candidate bounding boxes (either all annotated regions or object proposals), and a natural language query string.

**Output**: A scalar score for each candidate bounding box, used to rank them. The top-ranked box is returned as the retrieval result.

## Scoring recipe

```python
def compute_metrics(scores_list, gt_boxes, iou_thresh=0.5):
    correct_top1 = 0
    correct_top10 = 0
    for scores, gt in zip(scores_list, gt_boxes):
        ranked = np.argsort(scores)[::-1]
        top1_box = boxes[ranked[0]]
        top10_boxes = [boxes[i] for i in ranked[:10]]
        if iou(top1_box, gt) >= iou_thresh: correct_top1 += 1
        if any(iou(b, gt) >= iou_thresh for b in top10_boxes): correct_top10 += 1
    return correct_top1 / len(scores_list), correct_top10 / len(scores_list)
```

## Common pitfalls

- P@1 includes non-informative queries where bag-of-words baselines fail, while P@1-NR excludes them; mixing these up skews comparison.
- IoU threshold is strictly 50% overlap, not the more common 0.75 or 0.5 for detection.
- Kitchen dataset evaluation uses image-level features instead of bounding boxes due to dataset characteristics, requiring model adaptation.

## Evidence (verbatim from paper)

> Similar to [10], we evaluate with "P@1-NR" corresponding to non-random top-1 precision computed on the those informative results and "P@1" corresponding to top-1 precision on all cases including non-informative results, where random guess is used.

## Citation

```bibtex
@misc{hu2015natural,
  title={Natural Language Object Retrieval},
  author={Hu et al. (2015)},
  year={2015},
  note={arXiv:1511.04164}
}
```

- arXiv: 1511.04164