Pick tokenizer algorithm, vocab size, library for a given corpus and deployment target. Use when you need help with tokenizer picker.
Scanned 9/8/2026
Install to Claude Code
Skills Directorynpx -y skills add anubhavg-icpl/vibe --skill p5-tokenizer-picker --agent claude-codeInstalls into .claude/skills of the current project.
Are you the author of P5 Tokenizer Picker?
Add the live security badge to your README — it updates automatically with every re-scan.
[](https://www.skillsdirectory.com/skills/anubhavg-icpl-p5-tokenizer-picker)More formats (shields.io, HTML) on the badges page.
---
name: tokenizer-picker
description: Pick tokenizer algorithm, vocab size, library for a given corpus and deployment target. Use when you need help with tokenizer picker.
license: CC-BY-NC-SA-4.0
phase: 5
lesson: 19
metadata:
version: 1.0.0
tags: [nlp, tokenization]
---
Given a corpus (size, languages, domain) and deployment target (training from scratch / fine-tuning / API-compatible inference), output:
1. Algorithm. BPE, Unigram, or WordPiece. One-sentence reason.
2. Library. SentencePiece, HF Tokenizers, or tiktoken. Reason.
3. Vocab size. Rounded to nearest 1k. Reason tied to model size and language coverage.
4. Coverage settings. `character_coverage`, `byte_fallback`, special-token list.
5. Validation plan. Average tokens-per-word on held-out set, OOV rate, compression ratio, round-trip decode equality.
Refuse to train a character-coverage <0.995 tokenizer on corpora with rare-script content. Refuse to ship a vocab without a frozen `tokenizer.json` hash check in CI. Flag any monolingual tokenizer under 16k vocab as likely under-spec.
Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.
No comments yet. Be the first to comment!
