Language models
Fast inference
Like keeping ingredients ready before cooking the next dish, inference systems save repeated work. Language-model serving first processes the prompt, then generates tokens one at a time. Caching, batching, and lower precision change how much time and memory this takes.
How it works
- Measure prompt processing time, time per generated token, throughput, and memory under the intended workload.
- During prefill, process the prompt and store each layer’s keys and values.
- During decode, process the most recently appended token while reusing cached keys and values for earlier tokens.
- Batch active requests and manage cache allocation so completed requests free space for new ones.
- Evaluate lower precision or speculative decoding against both output quality and measured latency.
$$\text{KV bytes}=2BLTH_{\mathrm{KV}}d_hb\quad(B:\text{batch},\ L:\text{layers},\ T:\text{cached tokens},\ H_{\mathrm{KV}}:\text{KV heads},\ d_h:\text{head size},\ b:\text{bytes per value})$$
The knobs that matter
- batch size
- raises concurrent work but can increase waiting time and cache memory
- weight and cache precision
- fewer bytes reduce memory at a possible quality cost
- KV head count
- fewer shared key-value heads reduce cache size
- maximum context
- longer prefixes consume more cache per request
What interviewers ask
- What does the KV cache store, and why does it speed up decoding?
- How would you estimate KV-cache memory for a batch of requests?
- How do prefill and decode differ as hardware workloads?
- When can quantization or speculative decoding fail to improve latency?
Common mistakes
- Assuming KV caching removes the need to attend to earlier tokens.
- Reporting throughput alone when users care about first-token and inter-token latency.
- Counting only model weights when estimating serving memory.
When to use it
Use these techniques when serving autoregressive models under a latency or capacity target. Benchmark the actual workload because an optimization can move the bottleneck without improving user-visible speed.
Implement it (12 problems)
- Symmetric INT8 QuantizationFreeEasySolve
- Beam Search DecodingFreeMediumSolve