Mt Quality Estimation Eval

---
name: mt-quality-estimation-eval
description: Evaluates the ability of LLMs to predict human-assigned Direct Assessment (DA) scores for machine translation outputs. It probes how well different prompting strategies (zero-shot, CoT, few-shot) and input components (source, reference, error words) correlate with human judgments across various language pairs. Use when the user wants to benchmark on WMT QE / DA dataset (EN-DE, EN-MR, EN-ZH, ET-EN, NE-EN, RO-EN, RU-EN, SI-EN), or asks about evaluating this task. Reports Spearman $
ho$.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.03278
  bibtex_key: qian2024whatdo
  confidence: high
---

# mt-quality-estimation-eval

> What do Large Language Models Need for Machine Translation Evaluation? — Qian et al. (2024) (arXiv:2410.03278, 2024)

## What this evaluates

Evaluates the ability of LLMs to predict human-assigned Direct Assessment (DA) scores for machine translation outputs. It probes how well different prompting strategies (zero-shot, CoT, few-shot) and input components (source, reference, error words) correlate with human judgments across various language pairs.

## Datasets

- **WMT QE / DA dataset (EN-DE, EN-MR, EN-ZH, ET-EN, NE-EN, RO-EN, RU-EN, SI-EN)** — total ?; splits: test (-1); repo https://github.com/surrey-nlp/LLM4MT_eval

## Metrics

- `Spearman $
ho$` **(primary)** — range: [-1, 1]
  - Spearman rank correlation coefficient between the model's predicted quality scores and the true human-assigned DA scores (averaged across annotators). Measures monotonic relationship between predicted and human scores.

## Input / output format

**Input**: Source sentence, machine-translated output, and optionally a reference translation, error words, and annotation guidelines, formatted according to specific prompt templates (T1-T8).

**Output**: A single numerical quality score (DA score). Models must output a valid number; outputs without a score are discarded as 'dropped rows'.

## Scoring recipe

```python
def compute_spearman_rho(predictions, gold_scores):
    # Filter out any predictions that are not valid numbers
    valid_pairs = [(p, g) for p, g in zip(predictions, gold_scores) if is_valid_number(p)]
    pred_vals = [p for p, g in valid_pairs]
    gold_vals = [g for p, g in valid_pairs]
    # Compute Spearman rank correlation
    rho = scipy.stats.spearmanr(pred_vals, gold_vals).correlation
    return rho
```

## Common pitfalls

- LLMs frequently fail to generate a valid numerical score, requiring rows to be dropped before correlation calculation, which can bias results if the failure rate is high or non-random.
- High-resource language pairs (e.g., EN-DE, EN-ZH) exhibit skewed score distributions toward higher values, which can artificially lower Spearman correlation compared to medium/low-resource pairs.
- CoT prompting can degrade performance on smaller models (<10B parameters) while helping larger ones, making model size a critical confounding factor.

## Evidence (verbatim from paper)

> Table 3: Spearman $
ho$ correlation scores achieved by zero-shot inference using Templates 1-6 (T1-6) on various open-source LLMs for each language pair (LP). D -> rows dropped as LLM generated output without a score.

## Citation

```bibtex
@misc{qian2024whatdo,
  title={What do Large Language Models Need for Machine Translation Evaluation?},
  author={Qian et al. (2024)},
  year={2024},
  note={arXiv:2410.03278}
}
```

- arXiv: 2410.03278