Tokenizer Task Eval

---
name: tokenizer-task-eval
description: Evaluates how different tokenizer configurations (pre-tokenizer, fitting corpus, vocabulary size) affect downstream BERT performance on tasks requiring robustness vs. sensitivity to language variation. Use when the user wants to benchmark on AV (Authorship Verification), PAN, CORE, NUCLE, Dialect, GLUE, GLUE+typo, or asks about evaluating this task. Reports accuracy, F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.15343
  bibtex_key: wegmann2025tokenization
  confidence: high
---

# tokenizer-task-eval

> Tokenization is Sensitive to Language Variation — Wegmann et al. (2025) (arXiv:2502.15343, 2025)

## What this evaluates

Evaluates how different tokenizer configurations (pre-tokenizer, fitting corpus, vocabulary size) affect downstream BERT performance on tasks requiring robustness vs. sensitivity to language variation.

## Datasets

- **AV (Authorship Verification)** — total ?; splits: test (-1)
- **PAN** — total ?; splits: test (-1)
- **CORE** — total ?; splits: test (-1)
- **NUCLE** — total ?; splits: test (-1)
- **Dialect** — total ?; splits: test (-1)
- **GLUE** — total ?; splits: test (-1)
- **GLUE+typo** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances out of total instances.
- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, computed per task (macro or micro average as standard for these benchmarks).

## Input / output format

**Input**: Text instances from each dataset, tokenized using varying pre-tokenizers (e.g., NO, WS, _WS, LLAMA3, GPT2), fitting corpora (PMed, Wiki, Twitter), and vocabulary sizes (500, 4k, 32k, 64k, 128k), then fed into fine-tuned BERT models.

**Output**: Predicted class labels for each downstream task (e.g., authorship, dialect, grammatical error, NLI).

## Scoring recipe

```python
def score(predictions, gold, task):
    if task in ['AV', 'PAN', 'CORE', 'GLUE', 'GLUE+typo']:
        return accuracy_score(gold, predictions)
    elif task in ['NUCLE', 'Dialect']:
        return f1_score(gold, predictions, average='macro')
    # Average across tasks for AVG column
    # Significance tested via McNemar's test with Bonferroni correction
```

## Common pitfalls

- Assuming optimal tokenizer settings (e.g., vocabulary size) are consistent across all tasks; they vary by task type.
- Overlooking the impact of pre-tokenizer design, which the paper finds has the greatest influence on performance.
- Ignoring statistical significance; performance differences are often small and require McNemar's test with Bonferroni correction to validate.

## Evidence (verbatim from paper)

> | Model | AV (acc) ↑ | PAN (acc) ↑ | CORE (acc) ↑ | NUCLE (F1) ↑ | Dialect (F1) ↑ | AVG |

## Citation

```bibtex
@misc{wegmann2025tokenization,
  title={Tokenization is Sensitive to Language Variation},
  author={Wegmann et al. (2025)},
  year={2025},
  note={arXiv:2502.15343}
}
```

- arXiv: 2502.15343