Moverscore

---
name: moverscore
description: Automated evaluation of text generation quality by computing semantic distance between system outputs and human references using contextualized embeddings and Earth Mover's Distance (EMD). It probes a model's ability to capture meaning-based similarity rather than surface-level n-gram overlaps across machine translation, summarization, dialogue, and image captioning tasks. Use when the user has predictions and gold and needs to compute Pearson r.
metadata:
  skill_kind: metric
  source_arxiv: 1909.02622
  bibtex_key: [REDACTED]
  confidence: high
---

# moverscore

> MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance — Zhao et al. (2019) (arXiv:1909.02622, 2019)

## What this evaluates

Automated evaluation of text generation quality by computing semantic distance between system outputs and human references using contextualized embeddings and Earth Mover's Distance (EMD). It probes a model's ability to capture meaning-based similarity rather than surface-level n-gram overlaps across machine translation, summarization, dialogue, and image captioning tasks.

## Datasets

- **WMT 2017** — total 21000; splits: test (-1)
- **TAC-2008 / TAC-2009** — total 92; splits: test (-1)
- **BAGEL** — total 202; splits: test (-1)
- **SFHOTEL** — total 398; splits: test (-1)
- **MS-COCO** — total 5000; splits: test (-1)

## Metrics

- `Pearson r` **(primary)** — range: [-1, 1]
  - Computes the Pearson correlation coefficient between the metric's scores and human-assigned quality scores across all instances in a dataset.
- `Spearman ρ` — range: [-1, 1]
  - Computes the Spearman rank correlation coefficient between the metric's scores and human-assigned quality scores.

## Input / output format

**Input**: System-generated text and one or more human reference texts.

**Output**: A scalar Earth Mover's Distance score between the contextualized embedding distributions of the system and reference texts.

## Scoring recipe

```python
def compute_pearson_r(predictions, gold):
    n = len(predictions)
    mean_p = sum(predictions) / n
    mean_g = sum(gold) / n
    cov = sum((p - mean_p) * (g - mean_g) for p, g in zip(predictions, gold))
    std_p = (sum((p - mean_p)**2 for p in predictions) / n) ** 0.5
    std_g = (sum((g - mean_g)**2 for g in gold) / n) ** 0.5
    return cov / (std_p * std_g)
```

## Common pitfalls

- MoverScore outputs a distance (lower is better), but the evaluation reports correlation with human scores (higher is better). Confusing the directionality is common.
- The metric's performance heavily depends on the embedding model and fine-tuning task; using pre-trained BERT without fine-tuning yields significantly lower correlation than MNLI-finetuned BERT.
- Aggregation via p-means and n-gram granularity (unigram vs bigram) substantially impact results, and defaults are not explicitly standardized across all tasks.

## Evidence (verbatim from paper)

> We employ Pearson’s r and Spearman’s ρ to measure the correlation. We consider two variants of MoverScore: word mover and sentence mover, described below. In all language pairs, the best correlation is achieved by our word mover metrics that use a BERT pretrained on MNLI as the embedding generator and PMeans to aggregate the embeddings from different BERT layers.

## Citation

```bibtex
@misc{zhao2019moverscore,
  title={MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance},
  author={Zhao et al. (2019)},
  year={2019},
  note={arXiv:1909.02622}
}
```

- arXiv: 1909.02622