Multimodal Mt Eval

---
name: multimodal-mt-eval
description: Evaluates a model's ability to leverage visual context to improve machine translation, particularly for ambiguous verbs or long sentences with irrelevant text. It also assesses the quality of learned joint visual-text embeddings through an image retrieval task. Use when the user wants to benchmark on Multi30K, Ambiguous COCO, IKEA, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1808.08266
  bibtex_key: zhou2018visual
  confidence: high
---

# multimodal-mt-eval

> A Visual Attention Grounding Neural Model for Multimodal Machine Translation — Zhou et al. (2018) (arXiv:1808.08266, 2018)

## What this evaluates

Evaluates a model's ability to leverage visual context to improve machine translation, particularly for ambiguous verbs or long sentences with irrelevant text. It also assesses the quality of learned joint visual-text embeddings through an image retrieval task.

## Datasets

- **Multi30K** — total 31014; splits: train (29000), val (1014), test (1000)
- **Ambiguous COCO** — total 461; splits: test (461)
- **IKEA** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram precision with brevity penalty, computed over tokenized sentences.
- `METEOR` — range: percent
  - Harmonic mean of unigram precision and recall, with penalties for fragmentation and synonym matching.
- `Recall@K (R@K)` — range: percent
  - For each caption, find the top K nearest images in a shared embedding space via cosine similarity; R@K is the fraction of captions where the ground-truth image is in the top K.

## Input / output format

**Input**: Per instance: an image and its corresponding source-language caption (for translation); or an image-caption pair (for retrieval).

**Output**: Translation: target-language sentence. Retrieval: ranked list of candidate images.

## Scoring recipe

```python
# Translation metrics
preds = beam_search_decode(src_img, beam_size=12)
bleu_score = compute_bleu(gold_tgt, preds)
meteor_score = compute_meteor(gold_tgt, preds)

# Retrieval metric (Recall@K)
img_emb = encode_image(src_img)
txt_emb = encode_caption(src_txt)
sim_scores = cosine_similarity(txt_emb, all_img_embs)
top_k_idx = argsort(sim_scores, k=K)
recall_at_k = 1.0 if gold_img_idx in top_k_idx else 0.0
```

## Common pitfalls

- Comparing single-run results against ensemble baselines (e.g., 'Imagination') without accounting for the typical 1-2 point BLEU/METEOR boost from ensembling.
- Assuming visual grounding always improves translation; on short/simple captions like Multi30K, text-only models often match or slightly outperform multimodal ones.

## Evidence (verbatim from paper)

> We evaluate the performance of all models using BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2014). Then we can compute the recall rate of the paired image in the top K nearest neighbors, which is also known as R@K score.

## Citation

```bibtex
@misc{zhou2018visual,
  title={A Visual Attention Grounding Neural Model for Multimodal Machine Translation},
  author={Zhou et al. (2018)},
  year={2018},
  note={arXiv:1808.08266}
}
```

- arXiv: 1808.08266