Referring Expression Generation Eval

---
name: referring-expression-generation-eval
description: This benchmark evaluates a model's ability to generate referring expressions that enable humans to quickly and accurately identify a target object in an image. It prioritizes human comprehension speed and accuracy over purely semantic correctness, particularly for low-salience targets. Use when the user wants to benchmark on RefCOCO, RefCOCO+, RefCOCOg, RefGTA, or asks about evaluating this task. Reports R1-CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1811.12104
  bibtex_key: tanaka2018referring
  confidence: high
---

# referring-expression-generation-eval

> Generating Easy-to-Understand Referring Expressions for Target Identifications — Tanaka et al. (2018) (arXiv:1811.12104, 2018)

## What this evaluates

This benchmark evaluates a model's ability to generate referring expressions that enable humans to quickly and accurately identify a target object in an image. It prioritizes human comprehension speed and accuracy over purely semantic correctness, particularly for low-salience targets.

## Datasets

- **RefCOCO** — total ?; splits: test_a (-1), test_b (-1)
- **RefCOCO+** — total ?; splits: test_a (-1), test_b (-1)
- **RefCOCOg** — total ?; splits: val (-1)
- **RefGTA** — total ?; splits: test (-1)

## Metrics

- `CIDEr` — range: [0, 1]
  - Computes the average n-gram similarity between generated sentences and ground-truth references, weighted by TF-IDF scores.
- `Meteor` — range: [0, 1]
  - Measures translation quality based on unigram precision, recall, and a penalty for word order.
- `R1-CIDEr` **(primary)** — range: [0, 1]
  - A ranking-weighted variant of CIDEr. Weights for ground-truth sentences are calculated as $w(r_{ij}) = (rank(r_{ij}) \sum_{j} rank(r_{ij})^{-1})^{-1}$, where rank is derived from human comprehension accuracy and time. The final score is the weighted average similarity.
- `Comprehension Accuracy` — range: percent
  - The percentage of human annotators who correctly identify the target object given the generated sentence.

## Input / output format

**Input**: An image and a target object (specified by a bounding box or object ID).

**Output**: A natural language referring expression (sentence) describing the target object.

## Scoring recipe

```python
def compute_r1_cider(predictions, golds, human_ranks):
    scores = []
    for pred, gt_list, img_id in zip(predictions, golds, image_ids):
        weights = []
        for j, gt in enumerate(gt_list):
            rank = human_ranks[(img_id, j)]
            denom = sum(1/r for r in [human_ranks[(img_id, k)] for k in range(len(gt_list))])
            w = (rank * denom) ** -1
            weights.append(w)
        score = weighted_cider(pred, gt_list, weights)
        scores.append(score)
    return mean(scores)
```

## Common pitfalls

- Assuming higher CIDEr/Meteor scores always correlate with better human comprehension; the paper explicitly notes that models with higher comprehension accuracy do not always generate sentences with higher automatic metric scores.
- Ignoring the 'impossible to identify' option in human evaluation; annotators were allowed to select this box, and results are reported both including and excluding these cases ('All' vs 'All (selected)').

## Evidence (verbatim from paper)

> In our study, the ideal metric should assign a high score to a sentence that can be easily comprehended by humans correctly and quickly. While CIDEr calculates the average similarity between a generated sentence from an object $o_{i}$ and ground-truth sentences ${r_{i1},\cdots,r_{im}}$; we define the ranking-weighted CIDEr (R-CIDEr) which utilizes weighted similarity scores between them by the inverse of their rank.

## Citation

```bibtex
@misc{tanaka2018referring,
  title={Generating Easy-to-Understand Referring Expressions for Target Identifications},
  author={Tanaka et al. (2018)},
  year={2018},
  note={arXiv:1811.12104}
}
```

- arXiv: 1811.12104