Toksuite Eval

---
name: toksuite-eval
description: This benchmark evaluates the robustness of language model tokenizers against real-world input perturbations, including orthographic errors, script variations, homoglyphs, diacritics, and stylistic changes across five languages. It isolates the impact of tokenizer design by testing identical model architectures with different tokenization strategies. Use when the user wants to benchmark on TokSuite, or asks about evaluating this task. Reports relative performance drop.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.20757
  bibtex_key: altintas2025toksuite
  confidence: high
---

# toksuite-eval

> TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior — Altintas et al. (2025) (arXiv:2512.20757, 2025)

## What this evaluates

This benchmark evaluates the robustness of language model tokenizers against real-world input perturbations, including orthographic errors, script variations, homoglyphs, diacritics, and stylistic changes across five languages. It isolates the impact of tokenizer design by testing identical model architectures with different tokenization strategies.

## Datasets

- **TokSuite** — total ?; splits: test (-1); repo https://github.com/r-three/Tokenizers

## Metrics

- `relative performance drop` **(primary)** — range: other
  - Calculated as (Acc_can - Acc_pert) / Acc_can, where Acc_can is accuracy on canonical input and Acc_pert is accuracy on perturbed input. Lower values indicate greater robustness; negative values indicate performance improvement on perturbed inputs.

## Input / output format

**Input**: Perturbed or canonical multiple-choice questions (text + options) in English, Turkish, Italian, Farsi, or Chinese.

**Output**: Model's predicted answer choice (e.g., A, B, C, D) or generated text response.

## Scoring recipe

```python
def compute_relative_drop(acc_can, acc_pert):
    if acc_can == 0:
        return 0.0
    return (acc_can - acc_pert) / acc_can

# For each perturbation category:
acc_can = mean(correct_predictions on canonical inputs)
acc_pert = mean(correct_predictions on perturbed inputs)
metric = compute_relative_drop(acc_can, acc_pert)
```

## Common pitfalls

- Negative metric values indicate performance improvement on perturbed inputs, not a calculation error.
- The metric measures relative drop, so a value of 0.0 means identical performance, while 1.0 means complete failure on perturbed inputs.
- Zero-width characters and homoglyphs affect morphologically complex languages (e.g., Farsi) differently due to whitespace and ligature dependencies.

## Evidence (verbatim from paper)

> Values represent relative performance drop  $(rac{	ext{Acc}_{	ext{can}} - 	ext{Acc}_{	ext{pert}}}{	ext{Acc}_{	ext{can}}})$ ; lower values indicate greater robustness.

## Citation

```bibtex
@misc{altintas2025toksuite,
  title={TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior},
  author={Altintas et al. (2025)},
  year={2025},
  note={arXiv:2512.20757}
}
```

- arXiv: 2512.20757