Serving a Model: Batching, the Key-Value Cache and Quantisation
For autoregressive transformer serving, small-batch decoding is often memory-bandwidth limited. Batching can share weight reads, a key-value cache reuses past attention keys and values, and quantisation reduces precision. Benefits depend on workload, hardware and task quality.
What a learner can do afterwards
- Explain when small-batch decoding can be bandwidth bound and prompt processing or training can be compute bound
- Explain what the key-value cache stores and how its stored data grows with retained tokens at fixed model and precision
- State the trade quantisation makes and one way to measure whether it was worth it
1 · Read
Here, serving means running a trained autoregressive transformer to answer requests without updating its weights. First, prompt processing, called prefill, handles the input tokens. The basic decoding loop then chooses one next token for each request, extends its context and repeats. This lesson describes that one-token loop; other generation methods can use different workflows.
Small-batch decoding often spends much of its time reading weights and cached attention state from device memory. It can be memory-bandwidth bound: faster arithmetic alone may not help much. Large-batch training and prompt processing often do more arithmetic per weight read and can be compute bound. Larger decoding batches can also shift the limit toward compute. Measure the actual workload.
Batching processes several requests together, allowing weight reads to serve more than one request. This can improve throughput and reduce cost per request, but the gain is not guaranteed. Weights, per-request caches and temporary working memory must fit. Compute capacity and latency targets also limit useful batch size. Waiting to collect a batch can delay a request.
At each attention layer, a token's query is compared with keys to set weights for combining values. The key-value cache stores processed tokens' keys and values at each layer. New tokens reuse them but still compute attention over retained context. With model and precision fixed and no eviction or compression, stored key/value data is proportional to retained tokens: twice the tokens means twice the data. Total device memory also includes weights and working memory.
Quantisation stores weights at lower precision. It can reduce memory and improve throughput, but speed and task quality depend on the model, format, hardware and workload. Quality loss is not always small. Compare the setting with a baseline on the same hardware and task mix. Measure tokens per second, cost per request, latency and task quality. Accept a setting only if quality and latency meet set limits and the measured resource gain meets the team's goal.
Separate prompt processing from decoding. Test batching, cache capacity and quantisation under the actual workload, with memory, latency and task-quality limits.
Lesson, questions and flashcards about autoregressive transformer serving were written for this dataset.
Reading and flashcards about comparing model-serving configurations were written for this dataset.
More to read
Plan a serving comparison
Write down the decision before running a serving test. You might want lower cost per request while keeping task quality and latency within set limits. Compare a baseline with a candidate on the same hardware. Keep prompt lengths, output lengths and request arrivals comparable. A faster result on easier requests does not establish a gain for your workload.
Change batching and weight precision separately when you want to identify each change's effect. Record tokens per second, cost per request and latency, together with task-quality results. Check memory use too: weights, active requests' key-value caches and temporary working memory all need space. At fixed model and cache precision, retaining more tokens requires more stored key/value data.
Report the settings alongside the results. Include the model, hardware, batch policy, weight precision, cache precision and any context eviction or compression. Another team needs these details to interpret the comparison. Choose a configuration that meets the required quality and latency limits and delivers the measured resource benefit you set out to test.
2 · Watch
به انگلیسیTake it off screen
Where it sits
Optional connections
This opens up
Nothing builds on it yet.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.