Parallelism And Sharding

---
name: parallelism-and-sharding
description: "Plan ColossalAI tensor, pipeline, sequence, hybrid parallelism,
  ShardFormer model sharding, auto-parallel, and offload workflows."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Parallelism and Sharding

Use this sub-skill when the task is about mapping GPUs to tensor/pipeline/sequence/data/expert parallelism, using ShardFormer, choosing `ShardConfig`, planning pipeline schedules, or diagnosing topology and sharding errors.

## Route Here

- Explain 1D/2D/2.5D/3D tensor parallelism, pipeline parallelism, sequence parallelism, zero-bubble, and hybrid topologies.
- Decide `tp_size`, `pp_size`, `sp_size`, `zero_stage`, microbatching, and divisibility constraints for `HybridParallelPlugin`.
- Use `ShardConfig`, `ShardFormer`, `ModelSharder`, and model policy concepts.
- Understand experimental auto-parallel, activation checkpointing, solver dependencies, and NVMe/offload caveats.
- Diagnose unsupported model policies, fused-kernel flags, and sequence-parallel incompatibilities.

## Reroute

- Process launch and environment variables: use `../installation-and-launch/SKILL.md`.
- Concrete `Booster` training loops and plugin object use: use `../booster-training/SKILL.md`.
- Colossal-Inference tensor or patched parallel generation commands: use `../inference-and-serving/SKILL.md`.
- Application-specific MoE/LLaMA/Chat/Eval scripts: use `../application-recipes/SKILL.md`.

## Fast Start

1. Confirm world size and GPU count.
2. Choose the largest hard partition first: pipeline stages, tensor shards, sequence shards, expert shards, then data parallel remainder.
3. Ensure the product of selected non-data parallel sizes divides world size.
4. Use `HybridParallelPlugin` for combined training topology and ShardFormer policies for model graph rewriting.
5. Validate with a tiny model or config advisor before using a real LLM.

## References and Helpers

- `references/parallelism-guide.md` explains topology choices and common hybrid patterns.
- `references/shardformer.md` covers ShardFormer, `ShardConfig`, model families, and optimization flags.
- `references/auto-parallel-and-offload.md` describes experimental auto-parallel, activation checkpointing, and NVMe/offload caveats.
- `references/troubleshooting.md` maps topology, NCCL, policy, sequence-parallel, and optional-kernel failures.
- `scripts/parallelism_config_advisor.py` checks world-size divisibility and suggests a data-parallel remainder.
- `scripts/shardformer_model_matrix.py` prints a self-contained model-family support checklist.

## Operating Rules

- Do not hide topology decisions inside a training-loop answer. State `tp_size`, `pp_size`, `sp_size`, `ep_size`, and data-parallel remainder.
- Do not enable flash attention, fused normalization, JIT fused kernels, or FP8 without checking optional dependencies and hardware.
- Treat auto-parallel and solver examples as experimental unless the user's environment matches the documented dependency path.