Training Across Many Devices: Data, Model and Pipeline Parallelism
Replicated data parallelism copies model parameters and splits training examples. Tensor parallelism divides arrays within layers, while pipeline parallelism assigns layers to stages. Memory needs, communication costs and scheduling determine which combination fits a training run.
What a learner can do afterwards
- Match each parallelism strategy to what it splits and what it has to communicate
- Use model-state size and measured communication costs to choose a parallelism strategy
- Explain what a pipeline bubble is and one way to shrink it
1 · Read
A batch is a group of training examples. In replicated data parallelism, each device holds the full set of parameters (the numbers being learned) and processes part of the batch. Each copy computes gradients, which guide parameter updates. Devices combine gradients before applying the same update. A training step is this compute-and-update cycle.
Model parallelism splits a model across devices. Tensor parallelism splits tensors, or arrays of numbers, within layers and exchanges partial results during layer computation. Pipeline parallelism assigns whole layers to stages. It sends activations, the outputs of layers, forward across stage boundaries and sends activation gradients backward. Tensor and pipeline parallelism are both forms of model parallelism.
A pipeline bubble is idle stage time spent waiting for pipeline work, including during fill and drain. Split a batch into microbatches so stages can process different chunks at once. With balanced stages and enough microbatches, idle time can take a smaller share of runtime. It does not vanish, and tiny chunks can add overhead. In this synchronous setup, accumulate gradients across a batch's microbatches before applying its update.
Replicated copies cannot make parameters fit when the parameters alone exceed each device's memory. Tensor or pipeline splitting can distribute those parameters. Training also stores gradients, optimizer state (data used by the update method), and activations. Smaller local batches can reduce activation memory. Sharded data-parallel methods can distribute model state. Check which memory use is the limit.
Network bandwidth is data transferred per second. Slow links increase transfer time. Tensor splits exchange within-layer partial results; pipelines send activations and their gradients across boundaries; replicas synchronize gradients. Exclude plans whose peak memory per device exceeds capacity. Compare step times for the same batch and hardware, including compute, communication and waiting. Add separately timed parts only when they do not overlap.
Replicate parameters and split examples, or split tensors and layers. Measure memory and communication costs. Microbatches reduce the pipeline's idle-time share but leave some waiting.
Lesson, questions and flashcards about distributed training were written for this dataset.
2 · Watch
Take it off screen
Where it sits
Learn first
Nothing; this is a starting point.
Optional connections
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.