Training Config

---
name: training-config
description: "Integrate DeepSpeed training into PyTorch, author and validate
  config JSON/HJSON, plan launcher resources, choose ZeRO/offload settings, and
  handle checkpoint save/load/export safely."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# DeepSpeed Training Config

Use this sub-skill when a future agent needs to wire DeepSpeed into a PyTorch training loop, create or audit `ds_config.json`, build `deepspeed` launcher commands, select ZeRO stages/offload, or plan training checkpoint save/load/export. This sub-skill covers training runtime setup only; route inference-only kernel injection to `inference-injection`, pipeline/MoE/sequence parallel details to `parallelism-moe`, and build diagnostics to `ops-tooling`.

## Fast Path

1. Integrate the model with `deepspeed.initialize(...)`; call `deepspeed.init_distributed()` only when the script needs a distributed process group before `initialize`.
2. Author config through `references/configuration.md`, then run `scripts/validate_ds_config.py --world-size <ranks> ds_config.json` before launching.
3. Build resource filters and `--no_ssh` commands through `references/launcher-and-checkpointing.md`; preview filters with `scripts/launcher_resource_preview.py`.
4. Choose ZeRO settings with `references/configuration.md`, keeping deprecated aliases out of new configs.
5. Plan checkpoint calls so all ranks participate; for ZeRO-3 exports, enable `stage3_gather_16bit_weights_on_model_save` or use the generated `zero_to_fp32.py`/utility path after checkpointing.

## Bundled References

- `references/api-reference.md`: `deepspeed.initialize`, `init_distributed`, engine training calls, `DeepSpeedConfig`, and `DeepSpeedZeroConfig` field facts.
- `references/configuration.md`: batch-size contract, JSON/HJSON authoring, precision, optimizer/scheduler, ZeRO/offload choices, and safe validation workflow.
- `references/launcher-and-checkpointing.md`: launcher hostfile/include/exclude/no-SSH syntax plus checkpoint save/load/export guardrails.
- `references/troubleshooting.md`: common failures and fixes for config parsing, launcher filters, checkpoint hangs, ZeRO-3 export, and CUDA op probing.

## Bundled Scripts

- `scripts/validate_ds_config.py`: Parses JSON or HJSON when `hjson` is installed, detects duplicate JSON keys, checks batch-size consistency, and warns about selected ZeRO pitfalls without importing or launching DeepSpeed.
- `scripts/launcher_resource_preview.py`: Parses hostfile text plus `--include`/`--exclude` style filters and reports the selected resources or conflicts without launching training.

## Native Candidates

Safe native candidates to consult during verification are the config parser/dict tests and launcher parser tests. Broad GPU/distributed checkpoint matrices should be skipped by default unless the verifier has explicit safe hardware and time approval.