Wili 2018 Eval

---
name: wili-2018-eval
description: Evaluates the ability of models to correctly identify the language of monolingual text paragraphs. It probes language identification capabilities across a wide range of languages (235) with balanced representation. Use when the user wants to benchmark on WiLI-2018, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1801.07779
  bibtex_key: thoma2018wili
  confidence: high
---

# wili-2018-eval

> The WiLI benchmark dataset for written language identification — Thoma (2018) (arXiv:1801.07779, 2018)

## What this evaluates

Evaluates the ability of models to correctly identify the language of monolingual text paragraphs. It probes language identification capabilities across a wide range of languages (235) with balanced representation.

## Datasets

- **WiLI-2018** — total 1000; splits: test (1000)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Standard classification F1 score, calculated per language and averaged (typically micro-averaged for the overall benchmark score). Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2*P*R/(P+R).

## Input / output format

**Input**: A monolingual text paragraph (approximately 1,000 tokens).

**Output**: A single ISO 639-3 language code (or Wikipedia-specific code for some languages).

## Scoring recipe

```python
def compute_f1(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(predictions)
```

## Common pitfalls

- The dataset uses some non-standard or Wikipedia-specific language codes (e.g., 'map-bms', 'lzh') that may not match standard ISO 639-3 lists.
- Text is pre-normalized to Unicode NFKC form, so models must handle normalization or the benchmark assumes it.
- Results are reported per-language; overall performance should be micro-averaged across all 235 languages.

## Evidence (verbatim from paper)

> Lang	Prec	Recall	F1	Lang	Prec	Recall	F1

## Citation

```bibtex
@misc{thoma2018wili,
  title={The WiLI benchmark dataset for written language identification},
  author={Thoma (2018)},
  year={2018},
  note={arXiv:1801.07779}
}
```

- arXiv: 1801.07779