Training

---
name: training
description: "Use BERT-pytorch to instantiate the model, train on a tiny corpus,
  choose CPU or CUDA, and save checkpoints."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# training

Use this sub-skill when the task is about `bert`, `BERT`, `BERTLM`, `BERTTrainer`, model construction, device choice, training, or checkpoint saving.

## What this covers

- Instantiate `BERT` and `BERTLM`.
- Build a `BERTTrainer` around a `DataLoader`.
- Choose CPU or CUDA explicitly for a smoke run.
- Save checkpoints and understand the `.ep{epoch}` suffix.
- Diagnose hidden-size, attention-head, CUDA, and checkpoint-path errors.

## What to do first

- Build or load a valid vocab and corpus first; if those are missing, hand off to `../data-preparation/SKILL.md`.
- Use `scripts/train_smoke.py` for a safe one-epoch smoke run with explicit device selection.
- Use CPU smoke when you only need a deterministic quick check.
- Use CUDA smoke only when `torch.cuda.is_available()` is true and the host really has a compatible GPU setup.

## Read these references

- `references/training-workflow.md`: model/trainer flow, device selection, and checkpoint semantics.
- `references/troubleshooting.md`: CLI bool flags, hidden/head mismatch, GPU, memory, and save-path issues.
- `../data-preparation/SKILL.md`: if the corpus or vocab is the actual blocker.

## Related scripts

- `scripts/train_smoke.py`: build a tiny dataset, train one epoch, and save a checkpoint.
- `../../scripts/make_tiny_corpus.py`: create a deterministic tiny corpus fixture when you want to run the smoke script against a known input.

## Common outputs

- `BERT.forward()` returns the encoded sequence tensor before the language-model heads.
- `BERTLM.forward()` returns next-sentence and masked-language-model log probabilities.
- `BERTTrainer.save(epoch, file_path)` writes `file_path + ".ep{epoch}"`.
- `BERTTrainer` chooses `cuda:0` only when CUDA is available and requested.

## Boundary reminders

- Do not solve corpus layout or vocab serialization here.
- Do not bury device-selection guidance inside the data-preparation route.
- Keep the route focused on model instantiation, training, and checkpointing.