Tokenization

---
name: "tokenization"
description: "Routes GPT2-Chinese tokenizer selection, vocabulary rebuilding,
  and BERT, word-level, or BPE setup."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# Tokenization

Use this sub-skill when the task is about choosing, inspecting, or rebuilding the tokenizer and vocabulary stack for GPT2-Chinese.

## Read first

- `../references/workflows.md` for the tokenizer-related end-to-end flow.
- `../references/data-formats.md` for corpus and vocabulary file shapes.
- `../references/model-overview.md` for matching a tokenizer to a model config.
- `../references/cli-reference.md` for the CLI flags that switch tokenization modes.
- `../references/troubleshooting.md` and this sub-skill's troubleshooting file for path and dependency failures.
- `scripts/build_vocab.py` in this sub-skill when you need a safe vocabulary builder.

## What belongs here

- The default character/BERT tokenizer in `tokenizations/tokenization_bert.py`.
- The word-level tokenizer in `tokenizations/tokenization_bert_word_level.py`.
- The BPE helper in `tokenizations/bpe_tokenizer.py`.
- Vocabulary bundles such as `cache/vocab_small.txt`, `cache/vocab.txt`, `cache/vocab_seg.txt`, `cache/vocab_all.txt`, and `cache/vocab_guwen.txt`.
- Rebuilding a vocabulary from a JSON corpus.

## What does not belong here

- Training schedules, checkpoints, or perplexity evaluation belong in training.
- Prompted generation belongs in generation.
- Generic tokenizer theory without this repo's files and vocab bundles belongs elsewhere.

## How to route a tokenizer request

1. Decide whether you need the default path or an alternate tokenizer.
   - Default char/BERT mode: the usual choice.
   - Word-level mode: choose when segmentation matters more than raw character pieces.
   - BPE mode: choose when you already have encoder and merge files for a subword setup.
2. Match the vocabulary to the model.
   - Keep `--tokenizer_path` aligned with `config/model_config*.json` `vocab_size`.
   - Use `cache/vocab_small.txt` with the compact smoke config.
3. Decide whether you need a fresh vocabulary.
   - Use `scripts/build_vocab.py` for a safe corpus-to-vocab helper.
   - Treat the legacy `cache/make_vocab.py` as historical source material, not the preferred runtime path.
4. Decide whether you need the repo dictionary path.
   - The word-level tokenizer uses `thulac` and the repo's dictionary file.
   - Path handling matters because the tokenizer loads that dictionary at import time.

## Common decision points

- If the user only needs a model smoke check, do not rebuild the vocabulary; use the bundled small vocab and the tiny config.
- If the corpus is already segmented or you care about word boundaries, use the word-level path and rebuild the vocab from that corpus.
- If the user wants to generate with `--segment`, make sure the training and generation vocab were built from the same segmentation style.
- If the task mentions BPE but the encoder and merge files are missing, stop and ask for them before promising a run.

## Output expectations

- A vocabulary file with special tokens first and corpus tokens after that.
- A tokenizer choice that matches the checkpoint and config.
- Clear notes about any word-level or BPE prerequisites that were required.