Turkish Lm Eval

---
name: turkish-lm-eval
description: Evaluates a model's ability to capture statistical patterns and long-range dependencies in Turkish text. It tests generative probability estimation at both subword and character levels across news and Wikipedia domains. Use when the user wants to benchmark on trwiki-67, trnews-64, or asks about evaluating this task. Reports Perplexity (Ppl), Bits-per-character (Bpc).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.01215
  bibtex_key: safaya2022mukayese
  confidence: high
---

# turkish-lm-eval

> Mukayese: Turkish NLP Strikes Back — Safaya et al. (2022) (arXiv:2203.01215, 2022)

## What this evaluates

Evaluates a model's ability to capture statistical patterns and long-range dependencies in Turkish text. It tests generative probability estimation at both subword and character levels across news and Wikipedia domains.

## Datasets

- **trwiki-67** — total 67000000; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese
- **trnews-64** — total 64000000; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese

## Metrics

- `Perplexity (Ppl)` **(primary)** — range: [0, inf)
  - Derivative of negative log-likelihood: Ppl = exp(-1/n * sum(log p_theta(x_i|x_<i))). Evaluated on trwiki-67.
- `Bits-per-character (Bpc)` **(primary)** — range: [0, inf)
  - Entropy-based derivative of Nll used for character-level modeling on trnews-64. Lower is better.

## Input / output format

**Input**: Raw cased text (trwiki-67) or lightly pre-processed character-level text (trnews-64).

**Output**: Probability distribution over the next token or character in the sequence.

## Scoring recipe

```python
def score_lm(predictions, gold, dataset):
    n = len(gold)
    nll = -1/n * sum(log(p_theta(x_i | x_<i)) for i in range(n))
    if dataset == 'trwiki-67':
        return exp(nll)  # Ppl
    else:
        return nll / log2(2)  # Bpc (entropy-based derivative)
```

## Common pitfalls

- Ppl must be computed with the same token count or normalized, otherwise results are incomparable.
- Models must only be trained on the provided training split; using external data violates the benchmark protocol.
- Character-level vs subword-level tokenization changes vocabulary size and parameter counts, affecting direct model comparison.

## Evidence (verbatim from paper)

> Language models are trained on minimizing the negative log-likelihood (Nll) of the training set, and their performance is measured based on how well they can generalize on the test set: Nll(X_test) = -1/n sum log p_theta(x_i|x_test<i). Word or sub-word level language models are evaluated using the word perplexity (Ppl) metric, a derivative of Nll. On the other hand, character language models are evaluated using entropy-based Bits-per-character (Bpc) metric... Note that Ppl needs to be computed with the same count of tokens, otherwise it needs to be normalized

## Citation

```bibtex
@misc{safaya2022mukayese,
  title={Mukayese: Turkish NLP Strikes Back},
  author={Safaya et al. (2022)},
  year={2022},
  note={arXiv:2203.01215}
}
```

- arXiv: 2203.01215