Pretraining And Deepspeed

---
name: pretraining-and-deepspeed
description: "Prepare and preflight Baichuan-7B pretraining corpora, tokenizer
  placement, DeepSpeed host/config files, launch command rendering, and
  checkpoint expectations without executing training."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Baichuan-7B Pretraining and DeepSpeed

Use this sub-skill when the task is about Baichuan-7B pretraining setup, corpus sharding, `tokenizer.model` placement, DeepSpeed JSON/hostfile validation, rendering the training launch command, checkpoint output expectations, or troubleshooting a failed training launch.

Do **not** use this sub-skill for C-Eval/MMLU benchmark execution or architecture-only inference. Route architecture and model internals questions to `../architecture-and-loading/`; route shared package/API and install issues to the root skill's `../../references/api-reference.md` and `../../references/troubleshooting.md`.

## Operating contract

- Treat the original `train.py` as a **launch script**, not an importable API. It parses arguments and initializes distributed DeepSpeed at module import time.
- Use bundled helpers for safe preflight work:
  - `scripts/validate_training_inputs.py` checks corpus, tokenizer, hostfile, DeepSpeed JSON, and checkpoint path layout.
  - `scripts/render_deepspeed_command.py` renders a `deepspeed ... train.py ...` command without running it.
- Never execute DeepSpeed training unless the user explicitly asks to run it and provides suitable GPU/cluster resources, training data, tokenizer, and runtime environment.
- Keep full training claims bounded: bundled helpers validate inputs and render commands only; DeepSpeed training itself is not run by default.

## Fast routing

| User intent | Read first | Action |
|---|---|---|
| "Prepare Baichuan pretraining data" | `references/workflows.md` | Confirm UTF-8 shard files, shard count as a multiple of total ranks, and tokenizer placement; run `validate_training_inputs.py`. |
| "Validate DeepSpeed settings" | `references/configuration.md` | Parse JSON and hostfile; check ZeRO/bf16/micro-batch expectations; report cluster/resource assumptions. |
| "Render/train command" | `references/workflows.md` | Use `render_deepspeed_command.py`; show the command and warnings, do not launch by default. |
| "Where do checkpoints go?" | `references/workflows.md` | Explain `checkpoint_saving_path` and `Epoch-N` tag expectations from `model_engine.save_checkpoint(...)`. |
| "Training launch failed" | `references/troubleshooting.md` | Match the symptom to missing tokenizer/data, malformed hostfile/config, dependency pins, or GPU/cluster issues. |

## Required user inputs for preflight

Ask for or infer these paths from the user's active training workspace:

- Corpus directory, defaulting to `data_dir` in the Baichuan demo.
- SentencePiece tokenizer path, defaulting to `tokenizer.model`.
- DeepSpeed JSON config path, defaulting to `config/deepspeed.json`.
- DeepSpeed hostfile path, defaulting to `config/hostfile`.
- Training entrypoint path if rendering a command, commonly `train.py` in an active checkout or equivalent copied training script.
- Checkpoint output directory, defaulting to `checkpoints`.
- Expected total rank count when the hostfile is unavailable.

## Evidence distilled

This sub-skill is based on the Baichuan-7B training sections in the Chinese and English READMEs, `train.py`, `config/deepspeed.json`, `config/hostfile`, `scripts/train.sh`, and `requirements.txt`. Treat exact `deepspeed==0.9.2`, `torch==2.0.0`, `transformers==4.29.1`, `xformers==0.0.20`, and `sentencepiece==0.1.97` compatibility as a runtime environment concern; the bundled helpers validate layout/configuration and do not prove a full GPU training launch.