Tokenizers

---
name: tokenizers
description: "Router for VQ, VQGAN, Stable Diffusion VAE, and Consistency
  Decoder tokenizer training, finetuning, reconstruction, and round-trip
  checks."
metadata:
  disco-role: operating
disable-model-invocation: true
license: MIT
---

# Tokenizers

Use this sub-skill for image-tokenizer work that stays inside tokenizer and reconstruction behavior.

## Owns
- VQ tokenizer training and finetuning.
- Reconstruction through VQ, VQGAN, Stable Diffusion VAE, and Consistency Decoder checkpoints.
- Image/code round-trip sanity checks for saved tokenizer codes.
- Legacy checkpoint migration notes for VQGAN `.ckpt` to `.pth` conversion.

## Routes out
- Code extraction, dataset cache prep, and other preprocessing -> `data-preparation`.
- Class-conditional generation, sampling, serving, and evaluation -> `class-conditional`.
- Text-conditional generation and evaluation -> `text-conditional`.
- Checkpoint publishing and remote mutation -> excluded.

## Best entry points
- Training: `scripts/train_vq.sh`, `scripts/train_vq_finetune.sh`, `scripts/train_vq_finetune_continue.sh`
- VQ reconstruction: `scripts/reconstruct_vq.sh`
- VQ validation / input sanity: `scripts/validate_vq.sh`
- VQGAN reconstruction: `scripts/reconstruct_vqgan.sh`
- Stable Diffusion VAE reconstruction: `scripts/reconstruct_vae.sh`
- Consistency Decoder reconstruction: `scripts/reconstruct_consistency_decoder.sh`
- Code/image round trip: `scripts/check_image_codes.py`

## Read before answering
- `references/workflows.md`
- `references/cli-reference.md`
- `references/checkpoints.md`
- `references/troubleshooting.md`

## Fast routing rules
- If the request mentions code extraction, T5 extraction, or dataset cache layout, send it to `data-preparation`.
- If it asks for c2i or t2i generation, send it to the generation sub-skill instead of this one.
- If a checkpoint decode fails, first check the model family, codebook size, image size, and `pretrained_models` or diffusers layout.