Multilingual Eval

---
name: multilingual-eval
description: Evaluates the multilingual capabilities of LLMs across understanding, generation, reasoning, and instruction-following tasks in both high- and low-resource languages. It measures how well models comprehend instructions, translate, summarize, and perform commonsense reasoning across 100+ languages. Use when the user wants to benchmark on PAWS-X, FLORES-101, XL-Sum, XCOPA, Self-Instruct*, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.01771
  bibtex_key: lai2024llmsbeyondenglish
  confidence: high
---

# multilingual-eval

> LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback — Lai et al. (2024) (arXiv:2406.01771, 2024)

## What this evaluates

Evaluates the multilingual capabilities of LLMs across understanding, generation, reasoning, and instruction-following tasks in both high- and low-resource languages. It measures how well models comprehend instructions, translate, summarize, and perform commonsense reasoning across 100+ languages.

## Datasets

- **PAWS-X** — total ?; splits: test (-1)
- **FLORES-101** — total ?; splits: test (-1)
- **XL-Sum** — total ?; splits: test (-1)
- **XCOPA** — total ?; splits: test (-1)
- **Self-Instruct*** — total ?; splits: test (-1)

## Metrics

- `BLEU (SacreBLEU)` — range: percent
  - Case-sensitive detokenized BLEU score computed using SacreBLEU.
- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels or choices out of total instances.
- `ROUGE-1` — range: [0, 1]
  - Multilingual ROUGE-1 score measuring unigram overlap between generated and reference summaries/responses.

## Input / output format

**Input**: Task-specific prompts for paraphrase identification, machine translation, summarization, commonsense reasoning, and instruction following. Prompts are provided in Appendix C of the paper.

**Output**: Model-generated text (translation, summary, reasoning choice, or instruction response) in the target language.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task):
    if task == 'FLORES-101':
        return sacrebleu.corpus_bleu(predictions, [golds]).score
    elif task in ['XCOPA', 'PAWS-X']:
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task in ['XL-Sum', 'Self-Instruct*']:
        return rouge1_score(predictions, golds, lang='multilingual')
```

## Common pitfalls

- Subsampling test sets (200 samples/language for FLORES-101, 250 for XL-Sum) may not capture full benchmark variance.
- Zero-shot evaluation setting limits direct comparison with instruction-tuned baselines that use task-specific prompts.
- Self-Instruct* is a translated subset rather than the original English benchmark, complicating cross-lingual performance baselines.

## Evidence (verbatim from paper)

> We evaluate xLLMs-100 on five typical benchmarks including generation, reasoning, understanding and expert-written tasks that measure the multilingual capabilities of LLMs, including both high-resource and low-resource languages. For FLORES-101, we report case-sensitive detokenized BLEU with SacreBLEU... For the XCOPA and PAWS-X benchmarks, we utilize the accuracy score for evaluation. For the XL-Sum and Self-Instruct* benchmark, we report the multilingual ROUGE-1 score implemented by Lin (2004).

## Citation

```bibtex
@misc{lai2024llmsbeyondenglish,
  title={LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback},
  author={Lai et al. (2024)},
  year={2024},
  note={arXiv:2406.01771}
}
```

- arXiv: 2406.01771