Optimizers Training

---
name: optimizers-training
description: "Choose and use bitsandbytes optimizers in PyTorch training loops,
  including 8-bit, 32-bit, paged variants, StableEmbedding, GlobalOptimManager
  overrides, and optimizer-state checks."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# Optimizers Training

Use this sub-skill when a user asks how to train with `bitsandbytes.optim`, replace a PyTorch optimizer with an 8-bit or paged optimizer, keep specific parameters in 32-bit, use `StableEmbedding`, or validate optimizer state after a training step.

## Route First

- For direct quantized layers, `bitsandbytes.nn.Linear8bitLt`, `Linear4bit`, quantization functions, or low-level matmul APIs, use `../quantized-modules-functions/`.
- For Hugging Face `BitsAndBytesConfig`, QLoRA model loading, PEFT, Accelerate, Diffusers, or FSDP-QLoRA, use `../transformers-integrations/`.
- For install/import failures, backend library loading, `python -m bitsandbytes`, CUDA/ROCm/XPU compatibility, or native-kernel diagnostics, use `../installation-diagnostics/`.

## References

- Read `references/optimizer-api.md` for optimizer families, verified signatures, defaults, and state knobs.
- Read `references/optimizer-workflows.md` for drop-in replacement patterns, `min_8bit_size`, `StableEmbedding`, `GlobalOptimManager`, paged optimizers, and validation steps.
- Read `references/troubleshooting.md` when memory savings are missing, overrides do not apply, state dicts fail, CPU/GPU expectations mismatch, or paged optimizers do not behave as expected.

## Bundled Script

- Run `python sub-skills/optimizers-training/scripts/cpu-optimizer-smoke.py --optimizer adam8bit --steps 3` for a tiny deterministic CPU smoke that imports bitsandbytes, trains a toy model without downloads, and reports optimizer state dtypes/devices.
- Use `--optimizer adam32bit` to compare the same toy loop with a 32-bit bitsandbytes Adam optimizer.
- Use `--force-8bit-small-tensors` only as a diagnostic to lower `min_8bit_size` for the toy model; real training should usually keep the default unless memory analysis justifies changing it.

## Quick Decision Rules

- Start with `bnb.optim.Adam8bit` or `bnb.optim.AdamW8bit` as a drop-in replacement when optimizer-state memory is a bottleneck and parameters dominate memory use.
- Keep the default `min_8bit_size=4096` unless the user understands that smaller tensors are intentionally left in 32-bit for stability and minimal savings.
- Use `bnb.nn.StableEmbedding` for NLP embedding layers when training with 8-bit optimizer states.
- Use `GlobalOptimManager` when embeddings, layer norm, biases, or other unstable parameters should stay 32-bit while the rest of the model uses 8-bit states.
- Use paged optimizers only for supported accelerators and memory-pressure workflows; do not promise gains on CPU-only runs or when activations dominate memory.