Wmt17 Paraphrase Eval

---
name: wmt17-paraphrase-eval
description: Evaluates how well a model's predicted quality scores correlate with human judgments on machine translation output. It probes semantic equivalence and paraphrase detection capabilities in the context of MT evaluation. Use when the user wants to benchmark on WMT17, or asks about evaluating this task. Reports Pearson |r|.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2107.08251
  bibtex_key: [REDACTED]
  confidence: high
---

# wmt17-paraphrase-eval

> Generative Pretraining for Paraphrase Evaluation — Weston et al. (2021) (arXiv:2107.08251, 2021)

## What this evaluates

Evaluates how well a model's predicted quality scores correlate with human judgments on machine translation output. It probes semantic equivalence and paraphrase detection capabilities in the context of MT evaluation.

## Datasets

- **WMT17** — total ?; splits: test (-1)

## Metrics

- `Pearson |r|` **(primary)** — range: [0, 1]
  - Pearson correlation coefficient between model predictions and human judgment scores, reported as absolute value.
- `Kendall |τ|` — range: [0, 1]
  - Kendall rank correlation coefficient between model predictions and human judgment scores, reported as absolute value.

## Input / output format

**Input**: A pair of texts: a system-generated translation (or paraphrase) and a human reference translation.

**Output**: A single scalar score representing the predicted quality/paraphrase similarity.

## Scoring recipe

```python
def compute_metrics(predictions, human_scores):
    # predictions and human_scores are aligned lists of floats
    pearson_r = pearsonr(predictions, human_scores)[0]
    kendall_tau = kendalltau(predictions, human_scores)[0]
    return abs(pearson_r), abs(kendall_tau)
```

## Common pitfalls

- Report absolute values of correlation coefficients, as negative correlations are flipped to positive.
- Average results across source languages to match the paper's headline numbers; per-language scores differ.
- Distinguish the fine-tuning data (WMT15/16 English subsets) from the evaluation benchmark (WMT17).

## Evidence (verbatim from paper)

> The metrics reported are absolute Kendall $|	au|$ and Pearson $|r|$ averaged across each source language.

## Citation

```bibtex
@misc{weston2021generative,
  title={Generative Pretraining for Paraphrase Evaluation},
  author={Weston et al. (2021)},
  year={2021},
  note={arXiv:2107.08251}
}
```

- arXiv: 2107.08251