Multiloko Eval

---
name: multiloko-eval
description: Evaluates LLM multilingual knowledge and instruction-following across 31 languages using locally sourced, language-specific questions, while comparing performance on original versus machine-translated data. Use when the user wants to benchmark on MultiLoKo, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.10356
  bibtex_key: hupkes2025multiloko
  confidence: medium
---

# multiloko-eval

> MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages — Hupkes et al. (2025) (arXiv:2504.10356, 2025)

## What this evaluates

Evaluates LLM multilingual knowledge and instruction-following across 31 languages using locally sourced, language-specific questions, while comparing performance on original versus machine-translated data.

## Datasets

- **MultiLoKo** — total ?; splits: test (-1); repo https://github.com/facebookresearch/multiloko

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - Proportion of instances where the post-processed model output exactly matches the gold answer.

## Input / output format

**Input**: Language-specific knowledge questions in 31 languages, presented with either a 5-shot prompt (base models) or 0-shot prompt (chat models).

**Output**: Curt, precise answers (e.g., number, name, location). Base models require minimal post-processing (lowercase, strip punctuation); chat models require additional cleaning to remove words like 'answer' and handle language-specific deviations.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_clean = postprocess(pred)
        if pred_clean == gold:
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- Assuming machine-translated parallel data yields equivalent performance to locally sourced original data.
- Overlooking model-specific post-processing requirements, which can artificially inflate or deflate scores if not standardized across base and chat models.
- Ignoring language-specific instruction-following deviations (e.g., in English and Japanese) that require custom post-processing rules.

## Evidence (verbatim from paper)

> To facilitate automatic evaluation, we include an instruction to answer questions curtly and precisely, producing only a number/name/location/etc. ... Because base models are good at following the instructions, minimal postprocessing is needed: we only lowercase the output and strip punctuation.

## Citation

```bibtex
@misc{hupkes2025multiloko,
  title={MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages},
  author={Hupkes et al. (2025)},
  year={2025},
  note={arXiv:2504.10356}
}
```

- arXiv: 2504.10356