Training Across Many Devices: Data, Model and Pipeline Parallelism · seed 1 · A4, ink-friendly. The answer key prints on its own page for grown-ups.

Split the work: batch, tensors, or layers

Computing · Machine Learning · ages 23-24
Name ______________________   Date ____________

  1. Which strategy splits tensors inside layers?

    • Pipeline parallelism
    • Model (tensor) parallelism
    • Single-device training

  2. In data parallelism, what is copied and what is split?

    • The model is copied and the batch is split
    • The batch is copied and the model is split
    • The gradients are copied and the data is deleted

  3. Which strategy splits whole layers into stages?

    • Data parallelism
    • Tensor parallelism
    • Pipeline parallelism

  4. What is a pipeline bubble?

    • A memory leak inside one stage
    • A burst of unusually fast traffic
    • Idle stage time during pipe fill and drain

  5. What must data-parallel devices exchange after each step?

    • Stage activations only
    • Gradients, so all copies stay in step
    • Nothing at all

  6. A run fits in memory but its links are slow. Which split fits?

    • Tensor parallelism across everything
    • Data parallelism, splitting the model no further than needed
    • One pipeline stage per layer

  7. A model that fits on no device is fixed by adding more data-parallel copies.

    Circle one:   True   False

  8. How do microbatches shrink the pipeline bubble?

    • They keep the next chunk right behind the previous one, so stages stay busy
    • They remove all communication between stages
    • They merge every stage onto one device
LightMySky · lightmysky.comW1-mt__e-JwIxHek-s1

Answer key

For grown-ups. Fold this page away before handing over the rest.

Split the work: batch, tensors, or layers W1-mt__e-JwIxHek-s1

  1. Model (tensor) parallelism · Tensor splits cut inside layers and trade partial results each step.
  2. The model is copied and the batch is split · Every device holds the full model and runs part of the batch.
  3. Pipeline parallelism · Stages are whole-layer blocks passing activations across boundaries.
  4. Idle stage time during pipe fill and drain · Stages wait without work while chunks arrive or leave.
  5. Gradients, so all copies stay in step · Pooling gradients keeps every copy learning the same update.
  6. Data parallelism, splitting the model no further than needed · Data syncs once per step, while finer splits chatter constantly.
  7. False · Copies still exceed memory, so the model itself must be split.
  8. They keep the next chunk right behind the previous one, so stages stay busy · Small chunks flow nose to tail, leaving little idle time.
Worksheet · LightMySky