Recipe1mplus Retrieval Eval

---
name: recipe1mplus-retrieval-eval
description: Evaluates cross-modal retrieval between food images and cooking recipes. It probes a model's ability to align visual and textual representations in a shared embedding space to rank relevant recipes given an image, and vice versa. Use when the user wants to benchmark on Recipe1M+, or asks about evaluating this task. Reports medR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.06553
  bibtex_key: marin2018recipe1mplus
  confidence: high
---

# recipe1mplus-retrieval-eval

> Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images — Marín et al. (2018) (arXiv:1810.06553, 2018)

## What this evaluates

Evaluates cross-modal retrieval between food images and cooking recipes. It probes a model's ability to align visual and textual representations in a shared embedding space to rank relevant recipes given an image, and vice versa.

## Datasets

- **Recipe1M+** — total ?; splits: test (1000)

## Metrics

- `medR` **(primary)** — range: other
  - Median rank of the ground-truth item in the retrieved list. Lower values indicate better retrieval performance.
- `R@K` — range: percent
  - Recall at top K: the percentage of queries where the ground-truth item appears in the top K retrieved results. Higher values indicate better performance.

## Input / output format

**Input**: A food image (for im2recipe retrieval) or a cooking recipe text (for recipe2im retrieval).

**Output**: A ranked list of candidate recipes (for im2recipe) or images (for recipe2im) from the test set, ordered by cosine similarity in the shared embedding space.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    ranks = []
    for pred, gt in zip(predictions, gold):
        rank = pred.index(gt) + 1
        ranks.append(rank)
    medR = np.median(ranks)
    r_at_k = {k: np.mean([1 if r <= k else 0 for r in ranks]) for k in [1, 5, 10]}
    return medR, r_at_k
```

## Common pitfalls

- MedR is lower-is-better, while R@K is higher-is-better; confusing the direction leads to misinterpreting results.
- The official evaluation uses a fixed subset of 1,000 test pairs repeated 10 times, not the full dataset.
- Human baseline uses a 10-way multiple-choice setup, which is not directly comparable to open retrieval without adjustment.

## Evidence (verbatim from paper)

> We report median rank (MedR), and recall rate at top $K$ (R@K) for all the retrieval experiments. To clarify, R@5 in the im2recipe task represents the percentage of all the image queries where the corresponding recipe is retrieved in the top $5$, hence higher is better.

## Citation

```bibtex
@misc{marin2018recipe1mplus,
  title={Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images},
  author={Marín et al. (2018)},
  year={2018},
  note={arXiv:1810.06553}
}
```

- arXiv: 1810.06553