--- title: "Training Across Many Devices: Data, Model and Pipeline Parallelism" description: "Replicated data parallelism copies model parameters and splits training examples. Tensor parallelism divides arrays within layers, while pipeline parallelism assigns layers to stages. Memory needs, co" canonical: https://lightmysky.com/learn/computing/training-across-many-devices-data-model-and-pipeline-parallelism-mt__e-JwIxHek source: https://lightmysky.com/learn/computing/training-across-many-devices-data-model-and-pipeline-parallelism-mt__e-JwIxHek.md retrieved: 2026-11-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Training Across Many Devices: Data, Model and Pipeline Parallelism Replicated data parallelism copies model parameters and splits training examples. Tensor parallelism divides arrays within layers, while pipeline parallelism assigns layers to stages. Memory needs, communication costs and scheduling determine which combination fits a training run. Subject: Computing · Area: Machine Learning · Ages 23 to 24 Page: https://lightmysky.com/learn/computing/training-across-many-devices-data-model-and-pipeline-parallelism-mt__e-JwIxHek ## Ready when they can - Match each parallelism strategy to what it splits and what it has to communicate - Use model-state size and measured communication costs to choose a parallelism strategy - Explain what a pipeline bubble is and one way to shrink it ## Lesson: Split the work: batch, tensors, or layers A batch is a group of training examples. In replicated data parallelism, each device holds the full set of parameters (the numbers being learned) and processes part of the batch. Each copy computes gradients, which guide parameter updates. Devices combine gradients before applying the same update. A training step is this compute-and-update cycle. Model parallelism splits a model across devices. Tensor parallelism splits tensors, or arrays of numbers, within layers and exchanges partial results during layer computation. Pipeline parallelism assigns whole layers to stages. It sends activations, the outputs of layers, forward across stage boundaries and sends activation gradients backward. Tensor and pipeline parallelism are both forms of model parallelism. **Example.** A pipeline bubble is idle stage time spent waiting for pipeline work, including during fill and drain. Split a batch into microbatches so stages can process different chunks at once. With balanced stages and enough microbatches, idle time can take a smaller share of runtime. It does not vanish, and tiny chunks can add overhead. In this synchronous setup, accumulate gradients across a batch's microbatches before applying its update. *(drawing: A1 means stage A on microbatch 1. Each cell is one forward time slot. This example omits backward work.)* Replicated copies cannot make parameters fit when the parameters alone exceed each device's memory. Tensor or pipeline splitting can distribute those parameters. Training also stores gradients, optimizer state (data used by the update method), and activations. Smaller local batches can reduce activation memory. Sharded data-parallel methods can distribute model state. Check which memory use is the limit. Network bandwidth is data transferred per second. Slow links increase transfer time. Tensor splits exchange within-layer partial results; pipelines send activations and their gradients across boundaries; replicas synchronize gradients. Exclude plans whose peak memory per device exceeds capacity. Compare step times for the same batch and hardware, including compute, communication and waiting. Add separately timed parts only when they do not overlap. **Recap.** Replicate parameters and split examples, or split tensors and layers. Measure memory and communication costs. Microbatches reduce the pipeline's idle-time share but leave some waiting. *Lesson, questions and flashcards about distributed training were written for this dataset.* ## Practice 8 questions on this page, each with its working shown. ## Optional connections - [Message Passing and Asynchronous Work](https://lightmysky.com/learn/computing/message-passing-and-asynchronous-work-mt_twcpO-945r) ## Opens up - [Serving a Model: Batching, the Key-Value Cache and Quantisation](https://lightmysky.com/learn/computing/serving-a-model-batching-the-key-value-cache-and-quantisation-mt_Nv1-WLGIO4)