Flores200 Mt Eval

---
name: flores200-mt-eval
description: Evaluates multilingual machine translation quality across 60 languages and 234 translation directions. It specifically probes a model's ability to handle high-, medium-, and low-resource languages while mitigating directional degeneration in symmetric multi-way translation. Use when the user wants to benchmark on FLORES-200, or asks about evaluating this task. Reports COMET-22.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.07003
  bibtex_key: luo2025beyondenglish
  confidence: high
---

# flores200-mt-eval

> Beyond English: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs — Luo et al. (2025) (arXiv:2511.07003, 2025)

## What this evaluates

Evaluates multilingual machine translation quality across 60 languages and 234 translation directions. It specifically probes a model's ability to handle high-, medium-, and low-resource languages while mitigating directional degeneration in symmetric multi-way translation.

## Datasets

- **FLORES-200** — total 2500; splits: devtest (2500); HF `flores`

## Metrics

- `COMET-22` **(primary)** — range: percent (0-100)
  - A neural reference-based metric that uses a multilingual encoder (XLM-R) to compute contextual representations of source, reference, and hypothesis, followed by a cross-attention mechanism and a regression head to predict a quality score. Scores are scaled to a 0-100 range for reporting.

## Input / output format

**Input**: Source sentence in language X, optionally prefixed with explicit direction tags and a target-language separator.

**Output**: Target sentence in language Y.

## Scoring recipe

```python
import comet
model = comet.load_model('Unbabel/comet-22')
scores = []
for src, ref, hyp in zip(sources, references, hypotheses):
    score = model.predict({'src': src, 'ref': ref, 'hyp': hyp})
    scores.append(score * 100)
return sum(scores) / len(scores)
```

## Common pitfalls

- Evaluating on language pairs not supported by both the baseline and the proposed model, which skews cross-model comparisons.
- Overlooking directional asymmetry, as performance often differs significantly between forward (En/Zh→X) and reverse (X→En/Zh) translation directions.
- Ignoring the custom Chinese-Mongolian testset, which was created via native annotators to fill a gap in FLORES-200.

## Evidence (verbatim from paper)

> We evaluate on FLORES-200 devtest. We adopt COMET-22 as our primary evaluation metric, and report SacreBLEU in the Appendix. For fairness, evaluation is conducted on the intersection of language pairs supported by each baseline and LMT, covering Chinese-centric directions when applicable.

## Citation

```bibtex
@misc{luo2025beyondenglish,
  title={Beyond English: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2511.07003}
}
```

- arXiv: 2511.07003