Chapter 4: LLM inference
Greedy, beam, top-k and top-p decoding
Every chatbot and translator turns chances into words, one token at a time. The rule for that pick changes the text a lot. Interviewers ask when to use beam search or sampling, and ask you to code a top-p filter.
What you will learn
- Implement greedy and beam search
- Implement top-k and top-p (nucleus) filtering
- Choose a decoding strategy for a task (translation vs chat vs code)
Key terms
- token
- a piece of text the model writes, here one word
- decoding
- the rule that picks each next word
- greedy decoding
- a rule that always takes the most likely next word
Prerequisites
Take Parallelism: data, tensor, pipeline and ZeRO/FSDP first. It comes right before this lesson in the course.
Where this lesson picks up: Earlier in the course, the model gave a chance for every token in its list, and the chances added up to 1. Now the model must write: which token does it take?
What comes next
Sampling picks tokens by their chances: can we make the chances sharper or flatter before we roll?
Next lesson: Temperature and entropy: how sampling reshapes the distribution
More lessons in this part
- Temperature and entropy: how sampling reshapes the distribution 18 min
- Repetition, penalties, constrained decoding and determinism 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.