Multilingual European Eval

---
name: multilingual-european-eval
description: Evaluates cross-lingual LLM performance across 20 European languages by translating five established benchmarks (ARC, HellaSwag, TruthfulQA, GSM8K, MMLU) and measuring task accuracy on the localized prompts. Use when the user wants to benchmark on ARC, HellaSwag, TruthfulQA, GSM8K, MMLU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.08928
  bibtex_key: thellmann2024multilingual
  confidence: medium
---

# multilingual-european-eval

> Towards Multilingual LLM Evaluation for European Languages — Thellmann et al. (2024) (arXiv:2410.08928, 2024)

## What this evaluates

Evaluates cross-lingual LLM performance across 20 European languages by translating five established benchmarks (ARC, HellaSwag, TruthfulQA, GSM8K, MMLU) and measuring task accuracy on the localized prompts.

## Datasets

- **ARC** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **TruthfulQA** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard accuracy for multiple-choice tasks; exact-match or token-level accuracy for open-ended generation tasks, computed per language.

## Input / output format

**Input**: Translated benchmark prompts in 20 European languages, preserving original task structure and using XML tags for formatting consistency.

**Output**: Model-generated completions (multiple-choice selections or free-form text).

## Scoring recipe

```python
for lang in languages:
    for dataset in [ARC, HellaSwag, TruthfulQA, GSM8K, MMLU]:
        preds = model.generate(translated_prompts[lang])
        gold = translated_answers[lang]
        score = compute_accuracy(preds, gold)
return mean(scores)
```

## Common pitfalls

- Translation fidelity varies significantly across the 20 languages, affecting evaluation reliability.
- Prompt formatting must be strictly preserved using XML tags to avoid breaking model inference.
- Large models require careful multi-GPU parallelism setup (data/model/pipeline) to avoid OOM errors.

## Evidence (verbatim from paper)

> We translated five well-known datasets, ARC, HellaSwag, TruthfulQA, GSM8K, and MMLU from English into 20 European languages. These datasets encompass a mix of multiple-choice and open-ended generation tasks, each presenting unique translation challenges. The translations preserved the original structure of each task to ensure consistency across languages.

## Citation

```bibtex
@misc{thellmann2024multilingual,
  title={Towards Multilingual LLM Evaluation for European Languages},
  author={Thellmann et al. (2024)},
  year={2024},
  note={arXiv:2410.08928}
}
```

- arXiv: 2410.08928