What does the key-value cache store?
- The training dataset
- Past attention keys and values
- User passwords
What work remains once model weights are frozen for serving?
- Generating output tokens
- Learning new weights
- Labelling training data
A served model produces output one token at a time.
Circle one: True False
What puts a ceiling on that batch growth?
- Weight memory plus growing cached context per request
- The number of users online
- The training loss
Why does cost per request fall as the batch grows?
- Accuracy rises with batch size
- Weights shrink with batch size
- One round of weight reads serves many requests
How do you judge whether quantisation was worth it?
- By speed alone
- By throughput and cost gains against measured accuracy loss
- By model size alone
Serving and training share the same bottleneck, so one tuning fixes both.
Circle one: True False
Long-context traffic grows and batches must shrink. What explains the squeeze?
- Tokenizers slow down
- Training restarts nightly
- Cached context per request eats the memory batching needs
Serving tokens without recomputing the past W1-mt_Nv1-WLGIO4-s1
- Past attention keys and values · Stored keys and values spare earlier tokens from recompute.
- Generating output tokens · Serving is inference: read tokens so far and pick the next.
- True · Each pick extends the context for the next pick.
- Weight memory plus growing cached context per request · Weights and per-token caches together fill memory.
- One round of weight reads serves many requests · Batching amortises the slow memory reads across requests.
- By throughput and cost gains against measured accuracy loss · Both sides of the trade need numbers from your own tasks.
- False · Bandwidth limits serving while compute limits training.
- Cached context per request eats the memory batching needs · Stored context grows with request length, crowding out batch slots.