Emma 500 Eval

---
name: emma-500-eval
description: Evaluates massively multilingual language models on intrinsic next-word prediction, machine translation, text classification, math reasoning, and code generation across dozens of languages, with a specific focus on low-resource language performance and cross-lingual transfer capabilities. Use when the user wants to benchmark on Glot500-c, Parallel Bible Corpus (PBC), FLORES-200, SIB-200, Taxi-1500, MGSM, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.17892
  bibtex_key: ji2024emma500
  confidence: high
---

# emma-500-eval

> EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models — Ji et al. (2024) (arXiv:2409.17892, 2024)

## What this evaluates

Evaluates massively multilingual language models on intrinsic next-word prediction, machine translation, text classification, math reasoning, and code generation across dozens of languages, with a specific focus on low-resource language performance and cross-lingual transfer capabilities.

## Datasets

- **Glot500-c** — total ?; splits: test (-1)
- **Parallel Bible Corpus (PBC)** — total ?; splits: test (-1)
- **FLORES-200** — total ?; splits: test (-1)
- **SIB-200** — total ?; splits: test (-1)
- **Taxi-1500** — total ?; splits: test (-1)
- **MGSM** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram precision score with brevity penalty, calculated using the flores200 tokenizer to handle non-whitespace-delimited languages at the sub-word level. Evaluated with sacrebleu signature: nrefs:1—case:mixed—eff:no—tok:flores200—smooth:exp—version:2.4.2.
- `chrF++` — range: percent
  - Character n-gram F-score that combines precision and recall of character n-grams. Uses word order 2 and the signature: nrefs:1—case:mixed—eff:yes—nc:6—nw:2—space:no—version:2.4.2.
- `Negative Log-Likelihood (NLL)` — range: other
  - Sum of log probabilities of ground-truth tokens given the model's distribution, computed over a concatenated test set using a sliding-window approach. Chosen over length-normalized perplexity to ensure fair cross-model comparison despite different tokenization schemes.
- `Accuracy (ACC)` — range: [0, 1]
  - Proportion of correctly predicted class labels. For classification tasks, the model scores the next-token probability for each candidate category, and the category with the highest probability is selected as the prediction.
- `pass@k` — range: [0, 1]
  - Fraction of generated code solutions that pass all test cases. Evaluated with a generation pool of 50 samples per problem for k values of 1, 10, and 25.

## Input / output format

**Input**: Varies by task: intrinsic eval uses concatenated test text with sliding windows; MT and classification use 3-shot prompting with demonstrations from the dev set; math uses direct or Chain-of-Thought prompting; code generation uses problem descriptions with test-case-based execution prompts.

**Output**: Varies by task: probability distributions over tokens for intrinsic eval; translated text sequences for MT; discrete class labels for classification; step-by-step reasoning and final answers for math; executable code snippets for code generation.

## Scoring recipe

```python
def score(predictions, gold, task):
    if task == 'mt':
        bleu = sacrebleu.corpus_bleu(predictions, [gold], tokenize='flores200')
        chrf = sacrebleu.corpus_chrf(predictions, [gold], char_order=6, word_order=2)
        return bleu.score, chrf.score
    elif task == 'classification':
        correct = sum(1 for p, g in zip(predictions, gold) if p == g)
        return correct / len(gold)
    elif task == 'intrinsic':
        nll = -sum(math.log(p) for p in predictions)  # log-probs of gold tokens
        return nll
    elif task == 'code':
        passed = sum(1 for code in predictions[:k] if execute_test_cases(code))
        return passed / len(predictions[:k])
    return 0
```

## Common pitfalls

- Using length-normalized perplexity instead of raw NLL for intrinsic evaluation, which biases results against models with different tokenizers.
- Applying standard sentencepiece or spacy tokenizers to BLEU instead of the required flores200 tokenizer, leading to incorrect sub-word alignment for non-whitespace languages.
- Ignoring the language resource stratification (high/medium-high/medium/medium-low/low) when reporting results, which masks critical low-resource performance differences.

## Evidence (verbatim from paper)

> The performance is measured by BLEU and chrF++ implemented in sacrebleu. The BLEU score is calculated with the flores200 tokenizer applied to the texts and chrF++ uses word order 2. The choice of flores200 tokenization ensures that languages that do not have a whitespace delimiter can be evaluated at the (sub-)word level. For reproducibility, we attach the BLEU and chrF++ signatures.

## Citation

```bibtex
@misc{ji2024emma500,
  title={EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models},
  author={Ji et al. (2024)},
  year={2024},
  note={arXiv:2409.17892}
}
```

- arXiv: 2409.17892