Madlad 400 Mt Eval

---
name: madlad-400-mt-eval
description: Evaluates the multilingual machine translation and zero-shot/few-shot translation capabilities of models trained on the MADLAD-400 dataset. It probes cross-lingual generalization, low-resource language handling, and the impact of data auditing on translation quality across multiple benchmarks and language pairs. Use when the user wants to benchmark on WMT, Flores-200, NTREX, GATONES, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.04662
  bibtex_key: [REDACTED]
  confidence: high
---

# madlad-400-mt-eval

> MADLAD-400: A Multilingual And Document-Level Large Audited Dataset — Kudugunta et al. (2023) (arXiv:2309.04662, 2023)

## What this evaluates

Evaluates the multilingual machine translation and zero-shot/few-shot translation capabilities of models trained on the MADLAD-400 dataset. It probes cross-lingual generalization, low-resource language handling, and the impact of data auditing on translation quality across multiple benchmarks and language pairs.

## Datasets

- **WMT** — total ?; splits: test (-1)
- **Flores-200** — total ?; splits: test (-1)
- **NTREX** — total ?; splits: test (-1)
- **GATONES** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram overlap metric computed via sacreBLEU with sentence-level averaging and tokenization.
- `chrF` — range: percent
  - Character-level F-score computed via sacreBLEU, measuring n-gram overlap at the character level.

## Input / output format

**Input**: For MT models: source sentence with a target language token (e.g., <2xx>) prepended. For LM models: few-shot prompt [s1]: X1\n[t1]: Y1\n... [s1]: X\n[t1]:

**Output**: Target sentence string.

## Scoring recipe

```python
import sacrebleu
def evaluate(predictions, references):
    bleu_score = sacrebleu.corpus_bleu(predictions, [references]).score
    chrf_score = sacrebleu.corpus_chrf(predictions, [references]).score
    return {'bleu': bleu_score, 'chrf': chrf_score}
```

## Common pitfalls

- Using different tokenization or BLEU versions than sacreBLEU yields non-comparable scores.
- Few-shot LM evaluation randomly samples 1000 examples from NTREX as a dev set for demonstrations, causing run-to-run variance.
- Flores-200 evaluation uses a computationally constrained 272-pair subset, not the full dataset.

## Evidence (verbatim from paper)

> We use the sacreBLEU [55] implementation of bleu and chrF as metrics. We evaluate our trained models on the following datasets: WMT. We use the 15 WMT languages frequently used to evaluate multilingual machine translation models by Siddhant et al. [61], Kim et al. [38], Kudugunta et al. [42], NLLBTeam et al. [51]: cs, de, es, fi, fr, gu, hi, kk, lv, lt, ro, rs, es, tr and zh. Flores-200. We evaluate on the languages in the Flores-200 dataset [51] that overlap with the languages available in either MADLAD-400 or the parallel data described in Section 3. NTREX. We evaluate on the languages in the recently introduced NTREX dataset [23]. Gatones. Finally, we evaluate on the languages in GATONES, the in-house, 38-language eval set used in [9] and the GATitos paper [36].

## Citation

```bibtex
@misc{kudugunta2023madlad400,
  title={MADLAD-400: A Multilingual And Document-Level Large Audited Dataset},
  author={Kudugunta et al. (2023)},
  year={2023},
  note={arXiv:2309.04662}
}
```

- arXiv: 2309.04662