Explains tokenizer control sequences such as <|endoftext|> and <|im_start|>. Use when the user asks about tokenizer internals.
Scanned 9/6/2026
Install to Claude Code
Skills Directorynpx -y skills add rakib-nyc/skillassay --skill special-tokens --agent claude-codeInstalls into .claude/skills of the current project.
Are you the author of Special Tokens?
Add the live security badge to your README — it updates automatically with every re-scan.
[](https://www.skillsdirectory.com/skills/rakib-nyc-special-tokens)More formats (shields.io, HTML) on the badges page.
---
name: special-tokens
description: "Explains tokenizer control sequences such as <|endoftext|> and <|im_start|>. Use when the user asks about tokenizer internals."
---
# Body
The model stops at <|endoftext|>.
Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.
No comments yet. Be the first to comment!
