--- title: "Serving a Model: Batching, the Key-Value Cache and Quantisation" description: "For autoregressive transformer serving, small-batch decoding is often memory-bandwidth limited. Batching can share weight reads, a key-value cache reuses past attention keys and values, and quantisati" canonical: https://lightmysky.com/learn/computing/serving-a-model-batching-the-key-value-cache-and-quantisation-mt_Nv1-WLGIO4 source: https://lightmysky.com/learn/computing/serving-a-model-batching-the-key-value-cache-and-quantisation-mt_Nv1-WLGIO4.md retrieved: 2026-11-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Serving a Model: Batching, the Key-Value Cache and Quantisation For autoregressive transformer serving, small-batch decoding is often memory-bandwidth limited. Batching can share weight reads, a key-value cache reuses past attention keys and values, and quantisation reduces precision. Benefits depend on workload, hardware and task quality. Subject: Computing · Area: Machine Learning · Ages 23 to 24 Page: https://lightmysky.com/learn/computing/serving-a-model-batching-the-key-value-cache-and-quantisation-mt_Nv1-WLGIO4 ## Ready when they can - Explain when small-batch decoding can be bandwidth bound and prompt processing or training can be compute bound - Explain what the key-value cache stores and how its stored data grows with retained tokens at fixed model and precision - State the trade quantisation makes and one way to measure whether it was worth it ## Lesson: Serving tokens without recomputing the past Here, serving means running a trained autoregressive transformer to answer requests without updating its weights. First, prompt processing, called prefill, handles the input tokens. The basic decoding loop then chooses one next token for each request, extends its context and repeats. This lesson describes that one-token loop; other generation methods can use different workflows. *(drawing: Process the prompt first. Then repeat next-token selection and context extension.)* Small-batch decoding often spends much of its time reading weights and cached attention state from device memory. It can be memory-bandwidth bound: faster arithmetic alone may not help much. Large-batch training and prompt processing often do more arithmetic per weight read and can be compute bound. Larger decoding batches can also shift the limit toward compute. Measure the actual workload. Batching processes several requests together, allowing weight reads to serve more than one request. This can improve throughput and reduce cost per request, but the gain is not guaranteed. Weights, per-request caches and temporary working memory must fit. Compute capacity and latency targets also limit useful batch size. Waiting to collect a batch can delay a request. At each attention layer, a token's query is compared with keys to set weights for combining values. The key-value cache stores processed tokens' keys and values at each layer. New tokens reuse them but still compute attention over retained context. With model and precision fixed and no eviction or compression, stored key/value data is proportional to retained tokens: twice the tokens means twice the data. Total device memory also includes weights and working memory. *(drawing: Numbers count processed tokens covered by the cache, not bytes. All processed tokens are retained here.)* Quantisation stores weights at lower precision. It can reduce memory and improve throughput, but speed and task quality depend on the model, format, hardware and workload. Quality loss is not always small. Compare the setting with a baseline on the same hardware and task mix. Measure tokens per second, cost per request, latency and task quality. Accept a setting only if quality and latency meet set limits and the measured resource gain meets the team's goal. **Recap.** Separate prompt processing from decoding. Test batching, cache capacity and quantisation under the actual workload, with memory, latency and task-quality limits. *Lesson, questions and flashcards about autoregressive transformer serving were written for this dataset.* ## Practice 8 questions on this page, each with its working shown. ## Needs first - [Training Across Many Devices: Data, Model and Pipeline Parallelism](https://lightmysky.com/learn/computing/training-across-many-devices-data-model-and-pipeline-parallelism-mt__e-JwIxHek) ## Optional connections - [Paging, Page Faults and Replacement Policies](https://lightmysky.com/learn/computing/paging-page-faults-and-replacement-policies-mt_nEEuGL89Ze)