Image Text Retrieval Eval

---
name: image-text-retrieval-eval
description: Evaluates a model's ability to retrieve relevant images given a text query and vice versa. It probes cross-modal alignment and ranking capabilities under both standard test-set and large-scale candidate-pool settings. Use when the user wants to benchmark on Flickr30k, COCO, or asks about evaluating this task. Reports Recall@K (R@K).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.12105
  bibtex_key: chen2022hivlp
  confidence: high
---

# image-text-retrieval-eval

> HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrieval — Chen et al. (2022) (arXiv:2205.12105, 2022)

## What this evaluates

Evaluates a model's ability to retrieve relevant images given a text query and vice versa. It probes cross-modal alignment and ranking capabilities under both standard test-set and large-scale candidate-pool settings.

## Datasets

- **Flickr30k** — total 31000; splits: train (29000), val (1000), test (1000)
- **COCO** — total 123000; splits: train (114000), val (5000), test (5000)

## Metrics

- `Recall@K (R@K)` **(primary)** — range: percent
  - For each query, rank candidates by similarity score. R@K equals 1 if at least one ground-truth match appears in the top-K ranked candidates, averaged over all queries. AR is the mean of R@K across K ∈ {1, 5, 10, 20}.

## Input / output format

**Input**: An image and a text query (or vice versa), paired with a candidate pool of images/texts to rank.

**Output**: A ranked list of candidate images or texts based on cross-modal similarity scores.

## Scoring recipe

```python
def compute_recall_at_k(scores, gold_indices, k):
    hits = 0
    for s, gold in zip(scores, gold_indices):
        top_k = sorted(range(len(s)), key=lambda i: s[i], reverse=True)[:k]
        if any(g in top_k for g in gold):
            hits += 1
    return (hits / len(scores)) * 100

def compute_ar(recall_values):
    return sum(recall_values) / len(recall_values)
```

## Common pitfalls

- The paper reports Recall values as percentages (e.g., 54.3), not decimals.
- Large candidate settings (Flickr30k-full/COCO-full) use the combined train+val+test sets as the retrieval pool, which differs from standard benchmark splits.
- Results are reported separately for image-to-text and text-to-image directions; AR is computed per direction, not averaged across directions unless specified.

## Evidence (verbatim from paper)

> We measure image-text retrieval by recall@$K$ (R@$K$) and the average R@$K$ (AR) for all $K$ for both image-to-text retrieval and text-to-image retrieval tasks. All metrics are the higher the better.

## Citation

```bibtex
@misc{chen2022hivlp,
  title={HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrieval},
  author={Chen et al. (2022)},
  year={2022},
  note={arXiv:2205.12105}
}
```

- arXiv: 2205.12105