Wals Metalinguistic Eval

---
name: wals-metalinguistic-eval
description: Probes large language models' ability to recall and identify structural and grammatical properties of languages across diverse linguistic domains. It measures whether models have internalized typological facts from the World Atlas of Language Structures (WALS) by answering multiple-choice questions about specific language features. Use when the user wants to benchmark on WALS, WALS-100, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.02182
  bibtex_key: arcon2026metalinguistic
  confidence: high
---

# wals-metalinguistic-eval

> Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages — Arčon et al. (2026) (arXiv:2602.02182, 2026)

## What this evaluates

Probes large language models' ability to recall and identify structural and grammatical properties of languages across diverse linguistic domains. It measures whether models have internalized typological facts from the World Atlas of Language Structures (WALS) by answering multiple-choice questions about specific language features.

## Datasets

- **WALS** — total 76475; splits: train (-1), val (-1), test (-1); repo https://github.com/Oranzna/metalinguistic_benchmark
- **WALS-100** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/Oranzna/metalinguistic_benchmark

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy for multiple-choice QA: 1 if the model's predicted answer exactly matches the ground-truth WALS value, 0 otherwise. Averaged across all language-feature instances.

## Input / output format

**Input**: A multiple-choice question template instantiated for a specific language and WALS feature, accompanied by a fixed set of possible answer options derived from the feature's value categories.

**Output**: The model must select exactly one answer option from the provided list.

## Scoring recipe

```python
correct = 0
total = 0
for instance in dataset:
    model_pred = generate_answer(instance.prompt)
    if model_pred == instance.ground_truth:
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Splitting by feature rather than language means models may memorize feature templates rather than generalizing to unseen languages.
- Annotation sparsity in WALS can confound genuine cross-linguistic performance differences with missing data.
- Surface-level memorization of terminology-heavy value labels may inflate scores if questions are not carefully rephrased.

## Evidence (verbatim from paper)

> accuracy in identifying grammatical features from the World Atlas of Language Structures (WALS) being strongly correlated with digital language resources rather than linguistic universals.

## Citation

```bibtex
@misc{arcon2026metalinguistic,
  title={Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages},
  author={Arčon et al. (2026)},
  year={2026},
  note={arXiv:2602.02182}
}
```

- arXiv: 2602.02182