Serving a Model: Batching, the Key-Value Cache and Quantisation
Generation is memory-bandwidth bound and runs one token at a time, so serving is a different problem from training. Batching amortises the weight reads, a key-value cache stops the model recomputing the past, and lower-precision weights buy throughput against a small accuracy loss.
What a learner can do afterwards
- Say why serving is bandwidth bound while training is compute bound
- Explain what the key-value cache stores and how its size grows with the request
- State the trade quantisation makes and one way to measure whether it was worth it
1 · Read
Serving is a different problem from training. The weights are already learned and frozen, and all the work lies in generating output. The model reads the tokens so far, picks the next one, and repeats, producing output one token at a time.
Generation is limited by how fast weights can be read from memory rather than by raw compute, which is why serving is called bandwidth bound while training is compute bound. Batching amortises those weight reads across requests, so each request gets cheaper as the batch grows, until memory ceilings stop the growth.
The key-value cache stops the model recomputing the past. It stores past attention keys and values so earlier tokens never need rework. That stored context grows with every token in the request, which is one more pressure on the batch ceiling.
Lower-precision weights buy throughput against a small accuracy loss. Judge the trade by measuring both sides: tokens per second and cost per request against accuracy on your own tasks. Adopt the setting only when the measured gain beats the measured loss.
Batch the reads, cache the past, quantise with measurement, and respect the memory ceiling.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.