Multimodal Grounding Eval

---
name: multimodal-grounding-eval
description: Evaluates a model's ability to localize text phrases and referring expressions within images by generating bounding box coordinates, and conversely to generate text descriptions from given bounding boxes. It probes spatial reasoning, text-image alignment, and zero-shot generalization across different expression types. Use when the user wants to benchmark on Flickr30k Entities, RefCOCO, RefCOCO+, RefCOCOg, or asks about evaluating this task. Reports R@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.14824
  bibtex_key: peng2023kosmos2
  confidence: high
---

# multimodal-grounding-eval

> Kosmos-2: Grounding Multimodal Large Language Models to the World — Peng et al. (2023) (arXiv:2306.14824, 2023)

## What this evaluates

Evaluates a model's ability to localize text phrases and referring expressions within images by generating bounding box coordinates, and conversely to generate text descriptions from given bounding boxes. It probes spatial reasoning, text-image alignment, and zero-shot generalization across different expression types.

## Datasets

- **Flickr30k Entities** — total ?; splits: val (-1), test (-1)
- **RefCOCO** — total ?; splits: val (-1), testA (-1), testB (-1)
- **RefCOCO+** — total ?; splits: val (-1), testA (-1), testB (-1)
- **RefCOCOg** — total ?; splits: val (-1), test (-1)

## Metrics

- `R@1` **(primary)** — range: percent
  - Recall at top-1 generated bounding box. A prediction is counted as correct if its Intersection over Union (IoU) with the ground-truth box exceeds 0.5.
- `R@5` — range: percent
  - Recall at top-5 generated bounding boxes. If fewer than 5 boxes are generated, all available boxes are used for calculation.
- `R@10` — range: percent
  - Recall at top-10 generated bounding boxes. Uses all available boxes if fewer than 10 are generated.
- `Accuracy` — range: percent
  - For referring expression comprehension, accuracy is measured using only the first generated bounding box, correct if IoU > 0.5.

## Input / output format

**Input**: Image embedding followed by a grounding prompt. For phrase grounding: '<s><image> Image Embedding </image><grounding>...<p>{phrase}</p>' where preceding words provide context. For referring expression comprehension: '<p>{referring expression}</p>'.

**Output**: Model generates location tokens enclosed in '<box>...</box>' tags, which are parsed into bounding boxes. For generation tasks, it outputs natural language text descriptions.

## Scoring recipe

```python
def compute_recall(predictions, gold_boxes):
    correct = 0
    for pred_box in predictions[:10]:
        if pred_box is None:
            break
        iou = compute_iou(pred_box, gold_boxes)
        if iou > 0.5:
            correct += 1
            break
    return (correct / len(gold_boxes)) * 100
```

## Common pitfalls

- Using isolated phrases without preceding context increases ambiguity; the protocol explicitly requires including preceding words as context.
- Malformed or unconvertible location sequences (e.g., '<box><loc1></box>') are treated as negative samples and must be filtered out before scoring.
- RefCOCO/RefCOCO+ datasets use shorter expressions from a two-player game, which inherently leads to lower performance compared to RefCOCOg.

## Evidence (verbatim from paper)

> We obtain the location tokens in “<box>..</box>” from the model response and then covert it into bounding boxes. The generated bounding box is correct if its intersection over union (IoU) with the ground-truth bounding box is greater than 0.5. ... We report the R@1, R@5, and R@10 metrics, where R@1/5/10 means calculating the recall using the top 1/5/10 generated bounding boxes.

## Citation

```bibtex
@misc{peng2023kosmos2,
  title={Kosmos-2: Grounding Multimodal Large Language Models to the World},
  author={Peng et al. (2023)},
  year={2023},
  note={arXiv:2306.14824}
}
```

- arXiv: 2306.14824