Chapter 4: LLM inference

The KV cache and its memory

A KV cache keeps every past key and value, so each decode step computes only the new token's row. The price is memory, and it grows with layers, heads, length and batch. Interviewers ask you to size this cache from a model's settings and to find how many sequences fit on one GPU.

18 minInteractive lessonChapter 4: LLM inferenceLesson 5 of 12

What you will learn

  • Implement a KV cache for autoregressive decoding
  • Estimate KV-cache size from a model config, context and batch
  • Find the maximum batch that fits next to the weights

Key terms

KV cache
a store that keeps every past token's key and value
row
one token's key, or its value, in the cache table
decode step
one pass of the model that makes one new token
np.vstack
a NumPy call that puts a new row under a table

Prerequisites

Take FlashAttention: tiled online softmax and memory traffic first. It comes right before this lesson in the course.

Where this lesson picks up: You saw that decode reads all the model's parameters to make one token, so memory speed limits it. But each new token also attends to every earlier token: must the model compute their keys and values again at every step?

What comes next

The cache fits, so how fast does each user see each token, and how many users can one server serve?

Next lesson: Latency metrics and GPU sizing: TTFT, ITL, throughput, goodput

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.