Multimodal Mt Rl Eval

---
name: multimodal-mt-rl-eval
description: Evaluates a multimodal sequence-to-sequence model's ability to generate accurate translations conditioned on both source text and image features. It specifically probes whether reinforcement learning with BLEU-based rewards can mitigate exposure bias and improve translation quality over standard supervised maximum likelihood estimation. Use when the user wants to benchmark on WMT17 multimodal machine translation shared task, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1805.02356
  bibtex_key: qian2018multimodal
  confidence: high
---

# multimodal-mt-rl-eval

> Multimodal Machine Translation with Reinforcement Learning — Qian et al. (2018) (arXiv:1805.02356, 2018)

## What this evaluates

Evaluates a multimodal sequence-to-sequence model's ability to generate accurate translations conditioned on both source text and image features. It specifically probes whether reinforcement learning with BLEU-based rewards can mitigate exposure bias and improve translation quality over standard supervised maximum likelihood estimation.

## Datasets

- **WMT17 multimodal machine translation shared task** — total ?; splits: train (29000), val (1000), test (1071)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Computes n-gram precision up to order 4 between generated and reference translations, multiplied by a brevity penalty to penalize short outputs. Corpus-level BLEU is reported as the final metric.
- `Perplexity` — range: [0, ∞)
  - Measures how well the model predicts the reference sequence, defined as exp(mean(-log p(x))). Lower values indicate better predictive performance.

## Input / output format

**Input**: English source sentence concatenated with 2048-dimensional ResNet-50 image features.

**Output**: Target language sentence (German, French, or Czech) generated via greedy search.

## Scoring recipe

```python
def compute_corpus_bleu(preds, refs):
    precisions = [count_ngram_matches(preds, refs, n) for n in range(1, 5)]
    brevity_penalty = min(1.0, len(preds) / len(refs))
    return brevity_penalty * (prod(precisions) ** 0.25) * 100

def compute_perplexity(model, data):
    log_probs = [model.log_prob(x) for x in data]
    return exp(mean(log_probs))
```

## Common pitfalls

- The RL training loop optimizes using sentence-level BLEU rewards, but the final reported performance metric is corpus-level BLEU, which can yield different model rankings.
- Image features are fixed ResNet-50 outputs rather than end-to-end learned representations, potentially introducing noise from irrelevant objects in the images.
- Perplexity is reported alongside BLEU but is not the primary optimization target; models with lower perplexity do not always achieve higher BLEU scores.

## Evidence (verbatim from paper)

> We use BLEU score [9] (ngram of up to 4) as our major automatic evaluation metrics. BLEU score is the n-gram overlap between output translation and reference translation. And a brevity term is added to penalize short translations.

## Citation

```bibtex
@misc{qian2018multimodal,
  title={Multimodal Machine Translation with Reinforcement Learning},
  author={Qian et al. (2018)},
  year={2018},
  note={arXiv:1805.02356}
}
```

- arXiv: 1805.02356