---
title: "Training Across Many Devices: Data, Model and Pipeline Parallelism"
description: "A run stops fitting on one device well before the interesting model sizes. Data parallelism copies the model and splits the batch, model parallelism splits the tensors, and pipeline parallelism splits"
canonical: https://lightmysky.com/learn/computing/training-across-many-devices-data-model-and-pipeline-parallelism-mt__e-JwIxHek
source: https://lightmysky.com/learn/computing/training-across-many-devices-data-model-and-pipeline-parallelism-mt__e-JwIxHek.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Training Across Many Devices: Data, Model and Pipeline Parallelism

A run stops fitting on one device well before the interesting model sizes. Data parallelism copies the model and splits the batch, model parallelism splits the tensors, and pipeline parallelism splits the layers, each paying a different communication bill.

Subject: Computing · Area: Machine Learning · Ages 23 to 24
Page: https://lightmysky.com/learn/computing/training-across-many-devices-data-model-and-pipeline-parallelism-mt__e-JwIxHek

## Ready when they can

- Match each parallelism strategy to what it splits and what it has to communicate
- Say which strategy a memory-bound run needs and which a bandwidth-bound run needs
- Explain what a pipeline bubble is and one way to shrink it

## Lesson: Split the work: batch, tensors, or layers

Data parallelism copies the whole model to every device and splits the batch between copies. Each device runs its share and then all devices pool their gradients, so every copy stays in step. The communication bill is that gradient sync after each step.

When the model itself stops fitting on one device, you split the model instead. Model parallelism splits tensors inside layers and passes partial results between devices at every step, which talks constantly and wants fast links. Pipeline parallelism splits whole layers into stages and passes activations across stage boundaries, which talks less often but leaves stages idle while the pipe fills and drains.

**Example.** Picture four stages in a line with one big batch rolling through. The last stage sits idle until the first chunk arrives, and the first stage sits idle after it sends its chunk onward. Those idle gaps are the pipeline bubble. Cut the batch into eight microbatches and the stages stay busy, because the next chunk is always right behind the previous one.

**Tip.** Pick the split from your binding constraint. If the model cannot fit on one device, split the model with tensor or pipeline parallelism. If slow links are the limit, lean on data parallelism and split the model only as far as memory forces, since fine-grained tensor traffic punishes slow links most.

**Recap.** Copy the model and split the batch, or split the tensors or the layers, and keep the pipe full with microbatches.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Message Passing and Asynchronous Work](https://lightmysky.com/learn/computing/message-passing-and-asynchronous-work-mt_twcpO-945r)

## Opens up

- [Serving a Model: Batching, the Key-Value Cache and Quantisation](https://lightmysky.com/learn/computing/serving-a-model-batching-the-key-value-cache-and-quantisation-mt_Nv1-WLGIO4)