Chapter 4: LLM inference
Prefill vs decode on the roofline
You will be able to place a GPU job on a speed chart and say if memory or maths limits it. That tells you why a fast GPU still writes tokens slowly for one user, and which fixes help. Interviewers for LLM inference roles ask this question often.
What you will learn
- Compute arithmetic intensity and place an op on the roofline
- Classify prefill and decode as compute- or memory-bound
Key terms
- kernel
- one job that a GPU runs, such as a matrix multiply
- bandwidth β
- the bytes per second that memory can deliver to the chip
- peak compute Π
- the most FLOPs per second that the chip can do
- arithmetic intensity
- the FLOPs that a job does for each byte it moves
Prerequisites
Take Repetition, penalties, constrained decoding and determinism first. It comes right before this lesson in the course.
Where this lesson picks up: You changed how the model picks its next token, with penalties, grammar masks and controlled sampling. Those choices change what the model writes, but what sets how fast it reads a prompt and writes each token?
What comes next
Attention also moves many bytes. Next, you will ask if it can avoid the step where it writes its whole score row to memory and reads it back.
Next lesson: FlashAttention: tiled online softmax and memory traffic
More lessons in this part
- FlashAttention: tiled online softmax and memory traffic 24 min
- The KV cache and its memory 18 min
- Latency metrics and GPU sizing: TTFT, ITL, throughput, goodput 20 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.