Chapter 4: LLM inference
The KV cache and its memory
A KV cache keeps every past key and value, so each decode step computes only the new token's row. The price is memory, and it grows with layers, heads, length and batch. Interviewers ask you to size this cache from a model's settings and to find how many sequences fit on one GPU.
What you will learn
- Implement a KV cache for autoregressive decoding
- Estimate KV-cache size from a model config, context and batch
- Find the maximum batch that fits next to the weights
Key terms
- KV cache
- a store that keeps every past token's key and value
- row
- one token's key, or its value, in the cache table
- decode step
- one pass of the model that makes one new token
- np.vstack
- a NumPy call that puts a new row under a table
Prerequisites
Take FlashAttention: tiled online softmax and memory traffic first. It comes right before this lesson in the course.
Where this lesson picks up: You saw that decode reads all the model's parameters to make one token, so memory speed limits it. But each new token also attends to every earlier token: must the model compute their keys and values again at every step?
What comes next
The cache fits, so how fast does each user see each token, and how many users can one server serve?
Next lesson: Latency metrics and GPU sizing: TTFT, ITL, throughput, goodput
More lessons in this part
- Prefill vs decode on the roofline 18 min
- FlashAttention: tiled online softmax and memory traffic 24 min
- Latency metrics and GPU sizing: TTFT, ITL, throughput, goodput 20 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.