Model Parallelism

---
name: model-parallelism
description: Split a model that will not fit on one GPU across devices with tensor, pipeline, and expert parallelism, choosing each split by its communication cost. Use when weights or activations exceed a single card and you must shard without stalling on the interconnect.
---

# Model parallelism

When a model is too large for one GPU, you cut it apart, and every cut buys
communication. Tensor, pipeline, and expert parallelism split along different
axes, each with its own network cost and its own way of leaving GPUs idle.
Choose badly and the cards wait on an all-reduce or sit in a pipeline bubble,
so the decision is really about matching the split to your interconnect and
batch shape.

## Method

1. **Compute the memory shortfall before choosing anything.** Weight bytes are
   roughly parameters times dtype bytes: a 70B model in FP16 wants about 140 GB
   and will not fit one 80 GB H100. For training, add optimizer state and
   activations. The gap sets the minimum GPU count ahead of any strategy.
2. **Keep tensor parallelism inside one fast domain.** Tensor parallelism
   (Megatron style) splits each matrix multiply across GPUs and all-reduces
   twice per layer, so it demands high bandwidth. Hold the tensor-parallel
   group within a single NVLink node, usually 8 GPUs, and never stretch it
   over InfiniBand.
3. **Use pipeline parallelism to cross node boundaries.** Pipeline parallelism
   assigns whole layer ranges to different GPUs and passes activations point to
   point, cheap enough for inter-node links. Its cost is the pipeline bubble:
   the idle time at the head and tail of every batch.
4. **Cut the bubble with more microbatches.** The bubble fraction is about
   (stages - 1) / microbatches. Four stages with four microbatches waste near 43
   percent; sixteen microbatches bring that under 20 percent. Raise the count
   until the bubble is small or activation memory runs out.
5. **Reach for expert parallelism only in a mixture-of-experts model.** MoE
   routes each token to a few experts, so you scatter experts across GPUs
   and pay an all-to-all to shuffle tokens. It grows parameter count without
   growing per-token compute, but the all-to-all is bursty and punishes
   routing imbalance.
6. **Factor the strategies as a grid, not a heap.** Real runs combine them:
   tensor parallel within a node, pipeline parallel across nodes, data parallel
   over the whole. Write the factorization out, for example TP=8, PP=4, DP=2 for
   64 GPUs, so the product equals your total device count exactly.
7. **Measure the interconnect before you trust the plan.** Run an NCCL bandwidth
   test and confirm NVLink and InfiniBand hit their expected GB/s. A tensor
   parallel group that accidentally crosses a PCIe hop instead of NVLink can
   halve throughput while looking correctly configured on paper.

## Litmus tests

- Does the tensor-parallel group stay inside one NVLink node?
- Is the microbatch count high enough to keep the pipeline bubble under 20
  percent?
- Does TP times PP times DP equal your total GPU count exactly?
- Did an NCCL bandwidth test confirm each group runs over the link you assumed?

## Boundaries

This is about splitting one model across devices. Replicating a model that does
fit and overlapping gradient communication is distributed-training-scaling;
sizing and running a single serving replica is vllm-serving. The partition that
is optimal depends on your model shape and cluster topology, which is a matter
of measurement, not doctrine.