Chikha Po Eval

---
name: chikha-po-eval
description: Evaluates fundamental lexical comprehension and generation capabilities of multilingual LLMs across thousands of languages. It probes word-level translation, context-aware translation, translation-conditioned language modeling, and bag-of-words machine translation tasks to measure basic linguistic competence beyond high-resource languages. Use when the user wants to benchmark on ChiKhaPo, or asks about evaluating this task. Reports language score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.16928
  bibtex_key: [REDACTED]
  confidence: medium
---

# chikha-po-eval

> ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models — Chang et al. (2025) (arXiv:2510.16928, 2025)

## What this evaluates

Evaluates fundamental lexical comprehension and generation capabilities of multilingual LLMs across thousands of languages. It probes word-level translation, context-aware translation, translation-conditioned language modeling, and bag-of-words machine translation tasks to measure basic linguistic competence beyond high-resource languages.

## Datasets

- **ChiKhaPo** — total ?; splits: test (-1)

## Metrics

- `language score` **(primary)** — range: [0, 1]
  - Accuracy computed over a minimum of 100 words per language. Scores are aggregated across languages after applying dataset caps.

## Input / output format

**Input**: Word-level prompts for translation tasks (WT, WTWC) and translation-conditioned language modeling/bag-of-words MT tasks.

**Output**: Predicted target word or sequence per prompt.

## Scoring recipe

```python
scores = {}
for lang in languages:
    if task in [WT, WTWC]: limit to 300 words
    if task in [TCLM, BOW MT]: use 30% of data
    if len(gold) >= 100:
        scores[lang] = accuracy(predictions, gold)
```

## Common pitfalls

- Evaluation uses capped subsets (300 words/language for WT/WTWC, 30% for TCLM/BOW MT) rather than full datasets.
- Reported scores require a minimum of 100 words per language to be valid.
- Performance is strongly influenced by language resourcedness, family, and task type.

## Evidence (verbatim from paper)

> All reported language scores across are computed over a minimum of 100 words per language.

## Citation

```bibtex
@misc{chang2025chikha-po,
  title={ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models},
  author={Chang et al. (2025)},
  year={2025},
  note={arXiv:2510.16928}
}
```

- arXiv: 2510.16928