Multilingual Translation Eval

---
name: multilingual-translation-eval
description: Evaluates large language models' multilingual instruction-following and non-English-centric translation capabilities across diverse language pairs and prompting strategies. The protocol tests how model performance varies when prompts are provided in different languages (e.g., Chinese, Finnish, English) versus automatically translated prompts. Use when the user wants to benchmark on NTREX-128, or asks about evaluating this task. Reports ChrF.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.04155
  bibtex_key: luo2025gloteval
  confidence: medium
---

# multilingual-translation-eval

> GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models — Luo et al. (2025) (arXiv:2504.04155, 2025)

## What this evaluates

Evaluates large language models' multilingual instruction-following and non-English-centric translation capabilities across diverse language pairs and prompting strategies. The protocol tests how model performance varies when prompts are provided in different languages (e.g., Chinese, Finnish, English) versus automatically translated prompts.

## Datasets

- **NTREX-128** — total ?; splits: test (-1)

## Metrics

- `ChrF` **(primary)** — range: [0, 1]
  - Character n-gram F-score measuring the overlap between the generated translation and the reference translation at the character level. Higher values indicate better translation quality.

## Input / output format

**Input**: Source text paired with a prompt template. Prompts are either language-specific (e.g., Chinese, Finnish, English) or automatically translated into 134 languages using Microsoft Translator service.

**Output**: Target language translation text.

## Scoring recipe

```python
def compute_chrf(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        scores.append(chrf_score(pred, ref))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Performance differences may be influenced by the quality of prompt translation via Microsoft Translator rather than pure model capability.
- Efficiency metrics (throughput) conflate model inference speed with backend implementation differences (vLLM vs HF Transformers) and tokenization complexity.

## Evidence (verbatim from paper)

> For evaluation, we utilized NTREX-128, a multi-aligned benchmark containing parallel texts across 128 languages, which is supported in GlotEval. ... The results of our case study (Figure[4]) clearly demonstrate EMMA-500’s performance compared to Llama-2-7B in multilingual instruction following capabilities and non-English-centric translation tasks. Specifically, EMMA-500 shows consistently higher ChrF scores across most language pairs for all six translation directions.

## Citation

```bibtex
@misc{luo2025gloteval,
  title={GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2504.04155}
}
```

- arXiv: 2504.04155