Multilingual Lang Prof Eval

---
name: multilingual-lang-prof-eval
description: Evaluates large language models' multilingual capabilities across 100–200 languages by aggregating performance on translation, question answering, mathematics, and reasoning tasks. It tracks proficiency trends over time and correlates them with language speaker counts, GDP, and data availability. Use when the user wants to benchmark on Aggregated Multilingual Tasks (Translation, QA, Math, Reasoning), or asks about evaluating this task. Reports language proficiency scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.08538
  bibtex_key: [REDACTED]
  confidence: low
---

# multilingual-lang-prof-eval

> The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks — Pomerenke et al. (2025) (arXiv:2507.08538, 2025)

## What this evaluates

Evaluates large language models' multilingual capabilities across 100–200 languages by aggregating performance on translation, question answering, mathematics, and reasoning tasks. It tracks proficiency trends over time and correlates them with language speaker counts, GDP, and data availability.

## Datasets

- **Aggregated Multilingual Tasks (Translation, QA, Math, Reasoning)** — total ?; splits: test (-1)

## Metrics

- `language proficiency scores` **(primary)** — range: unspecified
  - Aggregated performance score across translation, question answering, mathematics, and reasoning tasks evaluated across 100–200 languages. Exact mathematical formulation is not provided in the text.

## Input / output format

**Input**: Not specified in the provided text.

**Output**: Not specified in the provided text.

## Scoring recipe

```python
def compute_language_proficiency_scores(predictions, gold):
    lang_scores = {}
    for lang in languages:
        task_scores = [score(task) for task in ['translation', 'qa', 'math', 'reasoning']]
        lang_scores[lang] = mean(task_scores)
    return aggregate(lang_scores)
```

## Common pitfalls

- Academic benchmark scores may not correlate with downstream application performance in a given language.
- Current version lacks cross-lingual trend analysis and has limited task coverage.
- Evaluation is auto-updating and dynamic, making static reproduction difficult without version pinning.

## Evidence (verbatim from paper)

> The graph confirms already known trends: English as the language with most resources employs the highest language proficiency scores. [Figure 2] shows language proficiency scores by different models over time, showcasing that Google models currently perform best.

## Citation

```bibtex
@misc{pomerenke2025ailanguageproficiencymonitor,
  title={The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks},
  author={Pomerenke et al. (2025)},
  year={2025},
  note={arXiv:2507.08538}
}
```

- arXiv: 2507.08538