Comet Mt Eval

---
name: comet-mt-eval
description: Probes the semantic fidelity and grammatical correctness of automated translation pipelines when converting English benchmarks into low-resource languages. It measures how well translation methods preserve task structure and downstream model performance consistency. Use when the user wants to benchmark on FLORES, WMT24++, MMLU, or asks about evaluating this task. Reports COMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.22207
  bibtex_key: yukhymenko2026recovered
  confidence: high
---

# comet-mt-eval

> Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets — Yukhymenko et al. (2026) (arXiv:2602.22207, 2026)

## What this evaluates

Probes the semantic fidelity and grammatical correctness of automated translation pipelines when converting English benchmarks into low-resource languages. It measures how well translation methods preserve task structure and downstream model performance consistency.

## Datasets

- **FLORES** — total 3001; splits: devtest (1012)
- **WMT24++** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)

## Metrics

- `COMET` **(primary)** — range: [0, 1]
  - Leverages multilingual pre-trained models to score translations by comparing source text, hypothesis, and reference. System-level scores are aggregated across all translations in the dataset.
- `LLM-as-a-judge win rate` — range: percent
  - Compares translated benchmarks against a baseline using a judge model (Gemini-2.5-Flash). Outputs win/draw/loss counts, aggregated as win rate or average improvement percentage.

## Input / output format

**Input**: Per instance: English source sentence, human reference translation, and model-generated hypothesis translation. For benchmark evaluation: translated questions/answers paired with baseline translations for pairwise comparison.

**Output**: COMET score (float 0–1) per instance, aggregated to system-level mean. For LLM-as-judge: categorical preference (win/draw/loss) per pair.

## Scoring recipe

```python
scores = []
for src, ref, hyp in dataset:
    scores.append(comet_model.predict(src, ref, hyp))
return sum(scores) / len(scores)

wins, draws, losses = 0, 0, 0
for pair in benchmark_pairs:
    verdict = judge_model.compare(pair.baseline, pair.proposed)
    if verdict == 'win': wins += 1
    elif verdict == 'draw': draws += 1
    else: losses += 1
return wins / (wins + draws + losses)
```

## Common pitfalls

- COMET scores are not absolute gold standards and can be biased by domain-specific training data or fail to capture stylistic preferences.
- Reference translations often contain multiple valid candidates, so high automated scores do not guarantee error-free or contextually perfect translations.
- LLM-as-a-judge evaluations may inherit model-family biases, as the judge model's architecture can influence preference scores.

## Evidence (verbatim from paper)

> We evaluate our proposed methods on English-Ukrainian translation using the COMET (Crosslingual Optimized Metric for Evaluation of Translation) metric. COMET leverages multilingual pre-trained models to assess translations by comparing source text, hypothesis, and reference, demonstrating higher correlation with human judgments than traditional metrics like BLEU or chrF++ *(Rei et al., [2020])*. We report COMET system-level scores (aggregated across all translations) in Table [1], using the Unbabel/XCOMET-XL model for reference-based quality estimation *(Guerreiro et al., [2023])*.

## Citation

```bibtex
@misc{yukhymenko2026recovered,
  title={Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets},
  author={Yukhymenko et al. (2026)},
  year={2026},
  note={arXiv:2602.22207}
}
```

- arXiv: 2602.22207