Serving tokens without recomputing the past

Computing · Machine Learning · ages 23-24

Name ______________________ Date ____________

What does a transformer key-value cache store?
- Attention keys and values from retained tokens
- Attention queries and scores for future tokens
- Model weights and gradients for later training
For ordinary autoregressive language-model serving with fixed weights, what work is needed to answer a text request?
- Update learned weights for each newly received prompt
- Process the prompt and generate the response tokens
- Create fresh labels for a new training dataset
In the basic autoregressive decoding loop taught here, each request adds one output token per iteration.
Circle one: True False
Which memory demands must fit when choosing a decoding batch size on a device?
- Weights, per-request caches and temporary working memory
- Weights and caches, excluding temporary working memory
- Weights and temporary buffers, excluding per-request caches
Why can batching improve throughput and reduce cost per request during decoding?
- Permanently shrink the model when requests join
- Remove all attention work from batched requests
- Share weight reads across requests in one batch
Which trade can weight quantisation make, and how should a team evaluate it against a baseline on the same hardware and task mix?
- Higher precision may save memory but reduce quality; measure throughput, cost, latency and quality
- Lower precision may save memory but reduce quality; measure throughput, cost, latency and quality
- Lower precision saves memory with no quality risk; measure throughput, cost, latency and quality
Prompt processing and large-batch training can be compute bound because they do less arithmetic per weight read than small-batch decoding.
Circle one: True False
A fixed model caches all processed tokens at fixed precision, with no eviction or compression. A request grows from 1,000 to 2,000 processed tokens. Ignoring allocation overhead, what happens to its stored key/value data?
- Its stored key/value data stays as before
- Its stored key/value data shrinks by half
- Its stored key/value data doubles in size

Answer key

For grown-ups. Fold this page away before handing over the rest.

Serving tokens without recomputing the past W1-mt_Nv1-WLGIO4-s1

Attention keys and values from retained tokens · The cache stores attention keys and values for processed tokens. New tokens still compute attention using the stored state.
Process the prompt and generate the response tokens · Serving processes the prompt and generates output using the trained weights. It does not update those weights for each request.
True · Each iteration chooses one next token for a request, extends that request's context and repeats.
Weights, per-request caches and temporary working memory · Device memory must hold model weights, the active requests' cached state and temporary working memory.
Share weight reads across requests in one batch · Batched requests can share weight reads. This does not shrink the weights or remove attention computation, and the benefit depends on the workload.
Lower precision may save memory but reduce quality; measure throughput, cost, latency and quality · Quantisation lowers weight precision and can save memory or improve throughput, but task quality may fall. Compare measured resources, latency and quality with the baseline, then check the required limits and resource goal.
False · The proposed reason is reversed. Prompt processing and large-batch training often do more arithmetic per weight read and can be compute bound. Small-batch decoding can instead be limited by reading weights and cached state.
Its stored key/value data doubles in size · At fixed model and precision, stored key/value data is proportional to retained tokens. Twice as many retained tokens means twice as much stored data.