Wmt19 Online Mt Selection Eval

---
name: wmt19-online-mt-selection-eval
description: Evaluates an online learning framework's ability to dynamically identify the highest-quality machine translation systems from an ensemble using minimal human feedback, and measures the sample efficiency (number of human assessments needed) to converge to the official top-performing systems. Use when the user wants to benchmark on WMT'19 News Translation, or asks about evaluating this task. Reports convergence_to_top3.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.13385
  bibtex_key: mendonca2021onlinelearningmt
  confidence: high
---

# wmt19-online-mt-selection-eval

> Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort — Mendonça et al. (2021) (arXiv:2105.13385, 2021)

## What this evaluates

Evaluates an online learning framework's ability to dynamically identify the highest-quality machine translation systems from an ensemble using minimal human feedback, and measures the sample efficiency (number of human assessments needed) to converge to the official top-performing systems.

## Datasets

- **WMT'19 News Translation** — total 7711; splits: test (7711)

## Metrics

- `convergence_to_top3` **(primary)** — range: iterations
  - The number of test segments processed (iterations) until the online algorithm's weighted ranking of MT systems matches the official WMT'19 top-3 systems for that language pair. Lower values indicate faster convergence with less human effort.

## Input / output format

**Input**: A source sentence, a reference translation, and automatic translations from N competing MT systems for a specific language pair, along with available human assessment scores or fallback metric scores.

**Output**: A probability/weight distribution over the N competing MT systems, updated sequentially after each segment's feedback is received.

## Scoring recipe

```python
def compute_convergence(predictions_weights, gold_top3, max_iters=7711):
    for t, weights in enumerate(predictions_weights):
        top_n = sorted(weights, key=weights.get, reverse=True)[:3]
        if set(top_n) == gold_top3:
            return t + 1
    return max_iters
```

## Common pitfalls

- Fallback strategies (human-zero, human-avg, human-comet) are used when human scores are missing, drastically altering the loss signal and convergence speed.
- The test set is shuffled once per run, and EXP3 requires averaging over 10 runs due to stochastic arm selection, making results sensitive to random seeds and shuffling.

## Evidence (verbatim from paper)

> The main questions addressed by our experiment are: (i) whether an online learning approach can give a greater weight to the top performing systems for each language pair according to the shared task's official ranking, and (ii) if so, how quickly (i.e., how many translations need to be assessed by human evaluators in order to find the best system).

## Citation

```bibtex
@misc{mendonca2021onlinelearningmt,
  title={Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort},
  author={Mendonça et al. (2021)},
  year={2021},
  note={arXiv:2105.13385}
}
```

- arXiv: 2105.13385