Pearson Correlation Coefficient

---
name: pearson-correlation-coefficient
description: Probes the ability of an automated metric to correlate with human judgments of translation quality. Specifically, it tests how well a model's predicted segment-level scores align with Direct Assessment (DA) human scores across multiple language pairs. Use when the user has predictions and gold and needs to compute Pearson correlation coefficient.
metadata:
  skill_kind: metric
  source_arxiv: 1907.12679
  bibtex_key: shimanaka2019machine
  confidence: high
---

# pearson-correlation-coefficient

> Machine Translation Evaluation with BERT Regressor — Shimanaka et al. (2019) (arXiv:1907.12679, 2019)

## What this evaluates

Probes the ability of an automated metric to correlate with human judgments of translation quality. Specifically, it tests how well a model's predicted segment-level scores align with Direct Assessment (DA) human scores across multiple language pairs.

## Datasets

- **WMT-2017 Metrics Shared Task** — total ?; splits: train (4824), val (536), test (3920)

## Metrics

- `Pearson correlation coefficient` **(primary)** — range: [-1, 1]
  - Computes the Pearson correlation coefficient between the metric's predicted scores and the Direct Assessment (DA) human evaluation scores for each translation segment.

## Input / output format

**Input**: Source sentence and machine-translated target sentence pair.

**Output**: A single continuous float score representing the predicted translation quality.

## Scoring recipe

```python
def compute_pearson(predictions, gold):
    n = len(predictions)
    mean_p = sum(predictions) / n
    mean_g = sum(gold) / n
    cov = sum((p - mean_p) * (g - mean_g) for p, g in zip(predictions, gold))
    std_p = (sum((p - mean_p)**2 for p in predictions) / n) ** 0.5
    std_g = (sum((g - mean_g)**2 for g in gold) / n) ** 0.5
    return cov / (std_p * std_g)
```

## Common pitfalls

- Evaluates only to-English language pairs, so results do not generalize to other directions.
- Uses Direct Assessment (DA) human scores as the ground truth, not BLEU or COMET.
- Segment-level evaluation requires averaging correlations across language pairs rather than pooling all segments globally.

## Evidence (verbatim from paper)

> We evaluated each metric using the Pearson correlation coefficient between the metric scores and the DA human scores.

## Citation

```bibtex
@misc{shimanaka2019machine,
  title={Machine Translation Evaluation with BERT Regressor},
  author={Shimanaka et al. (2019)},
  year={2019},
  note={arXiv:1907.12679}
}
```

- arXiv: 1907.12679