Parallelism Moe

---
name: parallelism-moe
description: "Design DeepSpeed pipeline parallelism, MoE layers, sequence
  parallelism, AutoSP, and activation checkpointing for training workflows."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Parallelism, MoE, and Checkpointing

Use this sub-skill when a task involves DeepSpeed training parallelism beyond basic ZeRO: pipeline parallel models, MoE layers, expert tensor parallelism, Ulysses or HF sequence parallelism, AutoSP, or activation checkpointing.

## Route by Task

- **Pipeline parallelism**: use `references/workflows.md#pipeline-parallelism` for `PipelineModule`, `LayerSpec`, `TiedLayerSpec`, `ProcessTopology`, and `PipelineEngine.train_batch()` / `eval_batch()` loops.
- **MoE and expert parallelism**: use `references/workflows.md#moe-and-expert-parallelism` for `deepspeed.moe.layer.MoE`, expert-parallel sizing, optimizer param groups, tuple outputs, PR-MoE, and expert tensor parallelism.
- **Sequence parallelism**: use `references/workflows.md#sequence-parallelism` for `DistributedAttention`, Ulysses HF registration, dataloader sharding, loss aggregation, and AutoSP.
- **Activation checkpointing**: use `references/workflows.md#activation-checkpointing` for `deepspeed.checkpointing.configure()` and checkpoint placement with model/pipeline parallelism.
- **API signatures**: run `scripts/inspect_parallelism_api.py --help`, then use it to print import availability and signatures without launching distributed jobs.

## Boundaries

- Stay here for `PipelineModule`, `LayerSpec`, `TiedLayerSpec`, `ProcessTopology`, `PipelineEngine`, MoE layers, MoE optimizer groups, sequence-parallel APIs, AutoSP, and activation checkpointing.
- Use the training-configuration sub-skill for basic ZeRO configuration, optimizer/offload tuning not specific to MoE, and standard `deepspeed.initialize()` config assembly.
- Use the inference/injection sub-skill for inference tensor parallelism and kernel injection.
- Use ops/tooling guidance for extension builds, fused op installation, NVMe, or environment build failures.

## References

- `references/api-reference.md` lists the installed signatures and source-backed API responsibilities.
- `references/workflows.md` gives compact design workflows and minimal patterns.
- `references/troubleshooting.md` maps common parallelism failures to likely fixes.