---
title: "Serving a Model: Batching, the Key-Value Cache and Quantisation"
description: "Generation is memory-bandwidth bound and runs one token at a time, so serving is a different problem from training. Batching amortises the weight reads, a key-value cache stops the model recomputing t"
canonical: https://lightmysky.com/learn/computing/serving-a-model-batching-the-key-value-cache-and-quantisation-mt_Nv1-WLGIO4
source: https://lightmysky.com/learn/computing/serving-a-model-batching-the-key-value-cache-and-quantisation-mt_Nv1-WLGIO4.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Serving a Model: Batching, the Key-Value Cache and Quantisation

Generation is memory-bandwidth bound and runs one token at a time, so serving is a different problem from training. Batching amortises the weight reads, a key-value cache stops the model recomputing the past, and lower-precision weights buy throughput against a small accuracy loss.

Subject: Computing · Area: Machine Learning · Ages 23 to 24
Page: https://lightmysky.com/learn/computing/serving-a-model-batching-the-key-value-cache-and-quantisation-mt_Nv1-WLGIO4

## Ready when they can

- Say why serving is bandwidth bound while training is compute bound
- Explain what the key-value cache stores and how its size grows with the request
- State the trade quantisation makes and one way to measure whether it was worth it

## Lesson: Serving tokens without recomputing the past

Serving is a different problem from training. The weights are already learned and frozen, and all the work lies in generating output. The model reads the tokens so far, picks the next one, and repeats, producing output one token at a time.

Generation is limited by how fast weights can be read from memory rather than by raw compute, which is why serving is called bandwidth bound while training is compute bound. Batching amortises those weight reads across requests, so each request gets cheaper as the batch grows, until memory ceilings stop the growth.

**Example.** The key-value cache stops the model recomputing the past. It stores past attention keys and values so earlier tokens never need rework. That stored context grows with every token in the request, which is one more pressure on the batch ceiling.

**Tip.** Lower-precision weights buy throughput against a small accuracy loss. Judge the trade by measuring both sides: tokens per second and cost per request against accuracy on your own tasks. Adopt the setting only when the measured gain beats the measured loss.

**Recap.** Batch the reads, cache the past, quantise with measurement, and respect the memory ceiling.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Training Across Many Devices: Data, Model and Pipeline Parallelism](https://lightmysky.com/learn/computing/training-across-many-devices-data-model-and-pipeline-parallelism-mt__e-JwIxHek)
- [Paging, Page Faults and Replacement Policies](https://lightmysky.com/learn/computing/paging-page-faults-and-replacement-policies-mt_nEEuGL89Ze)

## Opens up

- [Interpretability: Probes, Features and Circuits](https://lightmysky.com/learn/computing/interpretability-probes-features-and-circuits-mt_C8POKvRkq6)