Mtqe Generation Based Eval

---
name: mtqe-generation-based-eval
description: Evaluates machine translation quality estimation (MTQE) methods by measuring how well their segment-level scores correlate with human judgments across multiple language pairs. It specifically tests a generation-based paradigm where LLMs create reference translations instead of directly scoring outputs. Use when the user wants to benchmark on WMT22 Test Sets (8 language pairs), or asks about evaluating this task. Reports Spearman rank correlation (ρ).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.16129
  bibtex_key: [REDACTED]
  confidence: high
---

# mtqe-generation-based-eval

> LLMs Are Not Scorers: Rethinking MT Evaluation with Generation-Based Methods — Cui et al. (2025) (arXiv:2505.16129, 2025)

## What this evaluates

Evaluates machine translation quality estimation (MTQE) methods by measuring how well their segment-level scores correlate with human judgments across multiple language pairs. It specifically tests a generation-based paradigm where LLMs create reference translations instead of directly scoring outputs.

## Datasets

- **WMT22 Test Sets (8 language pairs)** — total ?; splits: test (-1)

## Metrics

- `Spearman rank correlation (ρ)` **(primary)** — range: [-1, 1]
  - Standard Spearman rank correlation coefficient measuring the monotonic relationship between predicted quality scores and human judgment scores.
- `Pearson correlation (r)` — range: [-1, 1]
  - Standard Pearson product-moment correlation coefficient measuring the linear relationship between predicted quality scores and human judgment scores.

## Input / output format

**Input**: Source sentence, machine-translated target sentence, and evaluation prompt.

**Output**: A continuous quality score (float) representing estimated translation quality, or a generated reference translation used for subsequent embedding-based similarity scoring.

## Scoring recipe

```python
def compute_spearman(pred_scores, human_scores):
    n = len(pred_scores)
    pred_ranks = [sorted(pred_scores).index(x) + 1 for x in pred_scores]
    human_ranks = [sorted(human_scores).index(x) + 1 for x in human_scores]
    d_sq = sum((p - h)**2 for p, h in zip(pred_ranks, human_ranks))
    return 1 - (6 * d_sq) / (n * (n**2 - 1))
```

## Common pitfalls

- Comparing reference-free methods against reference-based metrics, which introduces different semantic inputs and violates fair comparison paradigms.
- Assuming larger LLM parameters guarantee better scoring stability or accuracy; the paper shows smaller models (e.g., LLaMA-3-8B) can outperform larger/unstable ones.
- Using direct LLM scoring (regression) instead of the proposed generation-based reference creation, which yields significantly lower correlation.

## Evidence (verbatim from paper)

> Empirical results across 8 LLMs and 8 language pairs show significantly higher segment-level correlation with human judgments (up to +68% improvement) compared to direct scoring baselines, demonstrating that LLMs excel at fluent generation and semantic alignment when used for reference creation rather than direct regression. Table 2 reports results for ρ and r across UK-EN, CS-EN, RU-EN, and DE-EN.

## Citation

```bibtex
@misc{cui2025llmsarenotscorers,
  title={LLMs Are Not Scorers: Rethinking MT Evaluation with Generation-Based Methods},
  author={Cui et al. (2025)},
  year={2025},
  note={arXiv:2505.16129}
}
```

- arXiv: 2505.16129