Split the work: batch, tensors, or layers

Computing · Machine Learning · ages 23-24

Name ______________________ Date ____________

Which strategy divides tensors within model layers across devices?
- Pipeline parallelism across whole layers
- Tensor parallelism within model layers
- Replicated data parallelism across examples
In replicated data parallelism, what is copied and what is split?
- Copy model parameters; split training examples
- Copy training examples; split model parameters
- Copy model parameters; repeat every training example
Which description correctly gives both pipeline parallelism's partition and its transfers across stage boundaries?
- Split examples; send activations forward and activation gradients backward
- Split whole layers; send activation gradients forward and activations backward
- Split whole layers; send activations forward and activation gradients backward
What is a pipeline bubble?
- Memory lost when a stage discards every incoming activation
- Network time spent copying the full dataset to every stage
- Idle stage time spent waiting for pipeline work
In synchronous replicated data parallelism, what must happen before all copies apply the common parameter update?
- Send only activations before the common parameter update
- Combine gradients before the common update
- Apply separate local updates without combining gradients
Each device has 32 GB free. Plans list estimated peak memory and communication time measured in link tests for the same batch and hardware. Each plan also needs 40 ms of other work, with no overlap. Which fits and has the shortest predicted step?
- Full replicas: 48 GB per device; 10 ms communication
- Pipeline split: 28 GB per device; 25 ms communication
- Tensor split: 24 GB per device; 70 ms communication
A model's parameters need 30 GB per full replica. Each device has 24 GB free. Adding full replicated data-parallel copies makes the parameters fit on each device.
Circle one: True False
With balanced stages and a suitable schedule, how can splitting a batch into more microbatches reduce the pipeline bubble's share of runtime?
- Overlap microbatch work and reduce the idle-time share
- Remove every activation and gradient transfer between stages
- Guarantee that each stage stays busy for the entire training step

Answer key

For grown-ups. Fold this page away before handing over the rest.

Split the work: batch, tensors, or layers W1-mt__e-JwIxHek-s1

Tensor parallelism within model layers · Tensor parallelism divides arrays within layers. Model parallelism is the broader category that also includes pipelines.
Copy model parameters; split training examples · Each device has the full parameter set and processes its share of the training examples.
Split whole layers; send activations forward and activation gradients backward · Pipeline parallelism assigns whole layers to stages. Activations cross stage boundaries forward, and activation gradients cross backward.
Idle stage time spent waiting for pipeline work · A bubble is idle stage time caused by waiting for pipeline work. Filling and draining the pipeline create examples of this waiting.
Combine gradients before the common update · The copies compute gradients on different examples. Combining those gradients lets them apply the same update.
Pipeline split: 28 GB per device; 25 ms communication · Full replicas exceed the 32 GB capacity. Both split plans fit. Pipeline takes 40 + 25 = 65 ms, while tensor takes 40 + 70 = 110 ms. Pipeline has the shortest predicted step among feasible plans.
False · Each full replica still needs 30 GB, which exceeds the 24 GB available on its device. Adding full copies does not reduce that requirement.
Overlap microbatch work and reduce the idle-time share · Different stages can process different microbatches at once. This can reduce the share of time lost to waiting, but transfers and some idle time remain.