Chapter 4: LLM inference

Greedy, beam, top-k and top-p decoding

Every chatbot and translator turns chances into words, one token at a time. The rule for that pick changes the text a lot. Interviewers ask when to use beam search or sampling, and ask you to code a top-p filter.

16 minInteractive lessonChapter 4: LLM inferenceLesson 1 of 12

What you will learn

  • Implement greedy and beam search
  • Implement top-k and top-p (nucleus) filtering
  • Choose a decoding strategy for a task (translation vs chat vs code)

Key terms

token
a piece of text the model writes, here one word
decoding
the rule that picks each next word
greedy decoding
a rule that always takes the most likely next word

Prerequisites

Take Parallelism: data, tensor, pipeline and ZeRO/FSDP first. It comes right before this lesson in the course.

Where this lesson picks up: Earlier in the course, the model gave a chance for every token in its list, and the chances added up to 1. Now the model must write: which token does it take?

What comes next

Sampling picks tokens by their chances: can we make the chances sharper or flatter before we roll?

Next lesson: Temperature and entropy: how sampling reshapes the distribution

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.