Conditional Unigram Tokenization Eval

---
name: conditional-unigram-tokenization-eval
description: Evaluates a conditional unigram tokenizer's cross-lingual alignment quality and its impact on downstream machine translation and language modeling tasks. It measures intrinsic tokenization properties, alignment accuracy, and task-specific performance metrics. Use when the user wants to benchmark on NLLB, MultiParaCrawl, WMT2020, Flores, WMT2020 test set, or asks about evaluating this task. Reports chrF++.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.07824
  bibtex_key: [REDACTED]
  confidence: high
---

# conditional-unigram-tokenization-eval

> Conditional Unigram Tokenization with Parallel Data — Vico & Libovický (2025) (arXiv:2507.07824, 2025)

## What this evaluates

Evaluates a conditional unigram tokenizer's cross-lingual alignment quality and its impact on downstream machine translation and language modeling tasks. It measures intrinsic tokenization properties, alignment accuracy, and task-specific performance metrics.

## Datasets

- **NLLB** — total 51000000; splits: train (51000000)
- **MultiParaCrawl** — total 483000; splits: train (483000)
- **WMT2020** — total 60000; splits: train (60000)
- **Flores** — total ?; splits: test (-1)
- **WMT2020 test set** — total ?; splits: test (-1)

## Metrics

- `Parity` — range: ratio
  - Ratio of tokens produced by the target tokenizer to tokens produced by the reference source tokenizer. Lower values indicate better cross-lingual sequence length alignment.
- `Fertility` — range: ratio
  - Average number of tokens produced per word. Lower values (minimum 1.0) indicate that words remain coherent semantic units.
- `One-to-one` — range: [0, 1]
  - Proportion of source tokens that have exactly one aligned target token, which is also aligned to exactly one source token. Measured on the source side. Higher is better.
- `Unaligned` — range: [0, 1]
  - Portion of source tokens that are not aligned to any target tokens. Measured on the source sequence. Lower is better.
- `chrF++` **(primary)** — range: percent
  - Character n-gram F-score used for machine translation evaluation. Higher scores indicate better translation quality.
- `Perplexity per byte` — range: ratio
  - Language model perplexity normalized by the byte length of the text, allowing fair comparison across different vocabulary sizes. Lower is better.

## Input / output format

**Input**: Parallel source-target sentence pairs for tokenizer training and intrinsic evaluation; monolingual target sentences for language modeling evaluation.

**Output**: Tokenized sequences for intrinsic/alignment metrics; translated text for MT evaluation; next-token probability distributions for language modeling.

## Scoring recipe

```python
def compute_parity(pred_tokens, ref_tokens):
    return len(pred_tokens) / len(ref_tokens)

def compute_chrf(hypotheses, references):
    import sacrebleu
    return sacrebleu.corpus_chrf(hypotheses, references).score

def compute_fertility(pred_tokens, ref_words):
    return len(pred_tokens) / len(ref_words)

def compute_one_to_one(alignments):
    one_to_one = sum(1 for s, t in alignments if s == t == 1)
    return one_to_one / len(alignments)
```

## Common pitfalls

- Parity and Fertility scores are highly sensitive to vocabulary size and language family similarities, not just tokenizer quality.
- Alignment metrics (One-to-one, Unaligned) rely on an external aligner (Eflomal) and dev-set priors, which can introduce bias if the dev set distribution differs from the test set.
- Perplexity per byte normalizes by byte length, making it incomparable to standard token-level perplexity.

## Evidence (verbatim from paper)

> We assess tokenization quality using the following metrics: Parity (↓). This measures the ratio of tokens produced by our tokenizer in the target language to those produced by the reference tokenizer in the source language (Petrov et al., [2023]). Optimal tokenization should yield similar sequence lengths across languages. Fertility (↓). This measures the average number of tokens per word (Rust et al., [2021]). Lower fertility (minimum 1.0) indicates that words remain coherent semantic units. We evaluate models using chrF++ on Flores test sets (and WMT2020 test set for German-Upper Sorbian)... we use perplexity per byte to compare models with different vocabularies.

## Citation

```bibtex
@misc{vico2025conditional,
  title={Conditional Unigram Tokenization with Parallel Data},
  author={Vico & Libovický (2025)},
  year={2025},
  note={arXiv:2507.07824}
}
```

- arXiv: 2507.07824