Grefcoco Eval

---
name: grefcoco-eval
description: Tests a model's ability to ground natural language expressions that may refer to zero, one, or multiple objects in an image. The model must output a corresponding set of bounding boxes rather than a single box, evaluating its capacity for multi-target and no-target referring expression comprehension. Use when the user wants to benchmark on gRefCOCO, or asks about evaluating this task. Reports set-matching accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.16182
  bibtex_key: he2023grec
  confidence: medium
---

# grefcoco-eval

> GREC: Generalized Referring Expression Comprehension — He et al. (2023) (arXiv:2308.16182, 2023)

## What this evaluates

Tests a model's ability to ground natural language expressions that may refer to zero, one, or multiple objects in an image. The model must output a corresponding set of bounding boxes rather than a single box, evaluating its capacity for multi-target and no-target referring expression comprehension.

## Datasets

- **gRefCOCO** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/henghuiding/gRefCOCO

## Metrics

- `set-matching accuracy` **(primary)** — range: [0, 1]
  - Matches predicted bounding box sets to ground truth sets using an IoU threshold (typically 0.5). A prediction is correct if every predicted box matches a ground truth box within the threshold and set sizes align. Accuracy is the percentage of correctly matched expressions across the dataset.

## Input / output format

**Input**: An image and a natural language referring expression.

**Output**: A set of bounding boxes $B=\{b_i\}$, where each $b_i$ is a predicted bounding box. The set size ranges from 0 (no-target expression) to multiple (multi-target expression).

## Scoring recipe

```python
def compute_set_matching_accuracy(predictions, ground_truth, iou_threshold=0.5):
    correct = 0
    for pred_boxes, gt_boxes in zip(predictions, ground_truth):
        if len(pred_boxes) != len(gt_boxes):
            continue
        matched = 0
        used_gt = set()
        for p in pred_boxes:
            best_iou = 0
            best_idx = -1
            for idx, g in enumerate(gt_boxes):
                if idx not in used_gt:
                    iou = compute_iou(p, g)
                    if iou > best_iou:
                        best_iou = iou
                        best_idx = idx
            if best_iou >= iou_threshold:
                matched += 1
                used_gt.add(best_idx)
        if matched == len(gt_boxes):
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Models trained on classic REC datasets assume exactly one target and will incorrectly output a single box for multi-target or no-target expressions, leading to artificially low scores if not handled.
- Evaluating with single-box matching metrics instead of set-based matching leads to unfairly low scores for models that correctly identify multiple objects but output them in a different order or with slight coordinate variations.

## Evidence (verbatim from paper)

> In contrast to classic REC, which outputs a single bounding box for a referring expression, GREC aims to output a set of bounding boxes $B=\{b_{i}\}$, where each $b_{i}$ corresponds to an object among all the target objects referred by the given expression. The number of bounding boxes can range from 0 to multiple, depending on the given expression.

## Citation

```bibtex
@misc{he2023grec,
  title={GREC: Generalized Referring Expression Comprehension},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2308.16182}
}
```

- arXiv: 2308.16182