Language models

Fast inference

Like keeping ingredients ready before cooking the next dish, inference systems save repeated work. Language-model serving first processes the prompt, then generates tokens one at a time. Caching, batching, and lower precision change how much time and memory this takes.

Cached K and V3 earlier tokensNew tokencompute 1 new Q,K, VAttention1 query over 4keysAppend K and Vcache now has 4tokens
At one layer, reuse 3 cached key-value pairs and compute projections only for the new token.

How it works

  1. Measure prompt processing time, time per generated token, throughput, and memory under the intended workload.
  2. During prefill, process the prompt and store each layer’s keys and values.
  3. During decode, process the most recently appended token while reusing cached keys and values for earlier tokens.
  4. Batch active requests and manage cache allocation so completed requests free space for new ones.
  5. Evaluate lower precision or speculative decoding against both output quality and measured latency.

$$\text{KV bytes}=2BLTH_{\mathrm{KV}}d_hb\quad(B:\text{batch},\ L:\text{layers},\ T:\text{cached tokens},\ H_{\mathrm{KV}}:\text{KV heads},\ d_h:\text{head size},\ b:\text{bytes per value})$$

The knobs that matter

batch size
raises concurrent work but can increase waiting time and cache memory
weight and cache precision
fewer bytes reduce memory at a possible quality cost
KV head count
fewer shared key-value heads reduce cache size
maximum context
longer prefixes consume more cache per request

What interviewers ask

  • What does the KV cache store, and why does it speed up decoding?
  • How would you estimate KV-cache memory for a batch of requests?
  • How do prefill and decode differ as hardware workloads?
  • When can quantization or speculative decoding fail to improve latency?

Common mistakes

  • Assuming KV caching removes the need to attend to earlier tokens.
  • Reporting throughput alone when users care about first-token and inter-token latency.
  • Counting only model weights when estimating serving memory.

When to use it

Use these techniques when serving autoregressive models under a latency or capacity target. Benchmark the actual workload because an optimization can move the bottleneck without improving user-visible speed.

Implement it (12 problems)

Learn it

Interview questions about fast inference