Xtreme Eval

---
name: xtreme-eval
description: Evaluates zero-shot cross-lingual transfer of multilingual language models. Models are trained exclusively on English-labeled data and then tested on 40 typologically diverse languages across nine tasks spanning sentence classification, structured prediction, question answering, and sentence retrieval. Use when the user wants to benchmark on XTREME, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.11080
  bibtex_key: hu2020xtreme
  confidence: high
---

# xtreme-eval

> XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization — Junjie Hu et al. (2020) (arXiv:2003.11080, 2020)

## What this evaluates

Evaluates zero-shot cross-lingual transfer of multilingual language models. Models are trained exclusively on English-labeled data and then tested on 40 typologically diverse languages across nine tasks spanning sentence classification, structured prediction, question answering, and sentence retrieval.

## Datasets

- **XTREME** — total ?; splits: train (-1), test (-1); repo https://github.com/google-research/xtreme

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted labels out of total instances.
- `F1` — range: [0, 1]
  - Macro-averaged F1 score, computed as the harmonic mean of precision and recall across all classes or tokens.
- `exact-match (EM)` — range: [0, 1]
  - Exact match for question answering: 1 if the predicted answer span exactly matches the gold answer span, 0 otherwise.

## Input / output format

**Input**: Multilingual text sequences (sentences, paragraphs, or QA pairs) provided in English for training and in 40 target languages for testing.

**Output**: Task-specific predictions: class labels for XNLI/PAWS-X/Tatoeba, token tags for POS/NER, answer spans for XQuAD/MLQA/TyDiQA-GoldP, and relevance scores for BUCC.

## Scoring recipe

```python
def evaluate(predictions, gold, task):
    if task in ['XNLI', 'PAWS-X', 'Tatoeba']:
        return accuracy(predictions, gold)
    elif task in ['POS', 'NER', 'BUCC']:
        return f1_score(predictions, gold, average='macro')
    elif task in ['XQuAD', 'MLQA', 'TyDiQA-GoldP']:
        em = exact_match(predictions, gold)
        f1 = qa_f1(predictions, gold)
        return f1, em
    return None
```

## Common pitfalls

- Using target-language training data beyond the provided English labels violates the strict zero-shot cross-lingual transfer constraint.
- Directly comparing scores across languages for tasks with different test sets (POS, NER, MLQA, TyDiQA-GoldP) is invalid as noted by the authors.
- Failing to properly align answer spans when translating QA data for translate-train baselines leads to incomparable scores.

## Evidence (verbatim from paper)

> Concretely, pretrained multilingual representations are fine-tuned on English labelled data of an xtreme task. The model is then evaluated on the test data of the task in the target languages. Metrics: Acc., F1, F1/EM, F1, Acc.

## Citation

```bibtex
@misc{hu2020xtreme,
  title={XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization},
  author={Junjie Hu et al. (2020)},
  year={2020},
  note={arXiv:2003.11080}
}
```

- arXiv: 2003.11080