Chapter 4: LLM inference

Prefill vs decode on the roofline

You will be able to place a GPU job on a speed chart and say if memory or maths limits it. That tells you why a fast GPU still writes tokens slowly for one user, and which fixes help. Interviewers for LLM inference roles ask this question often.

18 minInteractive lessonChapter 4: LLM inferenceLesson 4 of 13

What you will learn

  • Compute arithmetic intensity and place an op on the roofline
  • Classify prefill and decode as compute- or memory-bound

Key terms

kernel
one job that a GPU runs, such as a matrix multiply
bandwidth β
the bytes per second that memory can deliver to the chip
peak compute Π
the most FLOPs per second that the chip can do
arithmetic intensity
the FLOPs that a job does for each byte it moves

Prerequisites

Take Repetition, penalties, constrained decoding and determinism first. It comes right before this lesson in the course.

Where this lesson picks up: You changed how the model picks its next token, with penalties, grammar masks and controlled sampling. Those choices change what the model writes, but what sets how fast it reads a prompt and writes each token?

What comes next

Attention also moves many bytes. Next, you will ask if it can avoid the step where it writes its whole score row to memory and reads it back.

Next lesson: FlashAttention: tiled online softmax and memory traffic

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.