Training

---
name: "training"
description: "Routes GPT2-Chinese corpus preprocessing, model training,
  checkpointing, and perplexity evaluation."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# Training

Use this sub-skill when the task is about training, fine-tuning, checkpointing, or perplexity evaluation in GPT2-Chinese.

## Read first

- `../references/workflows.md` for the end-to-end training and evaluation flow.
- `../references/cli-reference.md` for exact flags and defaults.
- `../references/data-formats.md` for the expected corpus and tokenized file shapes.
- `../references/model-overview.md` when choosing a config file or vocabulary bundle.
- `../references/troubleshooting.md` and this sub-skill's troubleshooting file for workflow-specific failures.
- `../../scripts/check_install.py` before the first training run.

## What belongs here

- Raw corpus preprocessing with `--raw`.
- `train.py` for the normal multi-article corpus path.
- `train_single.py` for one long source document or a single concatenated corpus.
- `eval.py` for perplexity estimation on a trained checkpoint.
- Checkpoint loading and continuation with `--pretrained_model`.
- TensorBoard logging and training-schedule options such as `--gradient_accumulation`, `--log_step`, `--stride`, and `--fp16`.

## What does not belong here

- Prompted text generation belongs in `sub-skills/generation/SKILL.md`.
- Tokenizer selection, vocabulary rebuilding, or BPE setup belongs in `sub-skills/tokenization/SKILL.md`.
- Generic Transformers questions belong in the broader model ecosystem, not this repo skill.

## How to route a training request

1. Match the corpus shape first.
   - JSON list of article strings: use `train.py`.
   - One long document: use `train_single.py`.
2. Match the config and vocab.
   - Use `config/model_config_test.json` for smoke checks.
   - Use `config/model_config_small.json` plus `cache/vocab_small.txt` for the usual compact run.
   - Keep `vocab_size` and `--tokenizer_path` aligned.
3. Decide whether to preprocess in the same run.
   - `--raw` tokenizes and shards the corpus before training.
   - If shards already exist, skip `--raw`.
4. Decide whether you are resuming from a checkpoint.
   - Pass `--pretrained_model` when you want to continue from `save_pretrained` output.
5. Keep the schedule options consistent.
   - `log_step` must divide `gradient_accumulation`.
   - `fp16` needs Apex and a matching CUDA stack.
6. For evaluation, create the output directory first if you want a persisted score file.

## Common decision points

- If the corpus is short and you only need a smoke check, use the test config and a tiny shard count.
- If you are training on a single long novel, prefer `train_single.py` over forcing `train.py` to split the text.
- If you need a perplexity number for a checkpoint, use `eval.py` after the checkpoint is saved.
- If the user is actually asking for generated samples, redirect to generation instead of training.

## Output expectations

- `output_dir/model_epochN/` after each epoch.
- `output_dir/final_model/` after the training loop completes.
- TensorBoard logs under the writer directory if configured.
- Tokenized shards under `tokenized_data_path` when `--raw` is used.