What does a transformer key-value cache store?

• Attention keys and values from retained tokens
• Attention queries and scores for future tokens
• Model weights and gradients for later training

For ordinary autoregressive language-model serving with fixed weights, what work is needed to answer a text request?

• Update learned weights for each newly received prompt
• Process the prompt and generate the response tokens
• Create fresh labels for a new training dataset

In the basic autoregressive decoding loop taught here, each request adds one output token per iteration.

Which memory demands must fit when choosing a decoding batch size on a device?

• Weights, per-request caches and temporary working memory
• Weights and caches, excluding temporary working memory
• Weights and temporary buffers, excluding per-request caches

Why can batching improve throughput and reduce cost per request during decoding?

• Permanently shrink the model when requests join
• Remove all attention work from batched requests
• Share weight reads across requests in one batch

Which trade can weight quantisation make, and how should a team evaluate it against a baseline on the same hardware and task mix?

• Higher precision may save memory but reduce quality; measure throughput, cost, latency and quality
• Lower precision may save memory but reduce quality; measure throughput, cost, latency and quality
• Lower precision saves memory with no quality risk; measure throughput, cost, latency and quality

Prompt processing and large-batch training can be compute bound because they do less arithmetic per weight read than small-batch decoding.

A fixed model caches all processed tokens at fixed precision, with no eviction or compression. A request grows from 1,000 to 2,000 processed tokens. Ignoring allocation overhead, what happens to its stored key/value data?

• Its stored key/value data stays as before
• Its stored key/value data shrinks by half
• Its stored key/value data doubles in size