Chapter 4: LLM inference
Repetition, penalties, constrained decoding and determinism
Models sometimes repeat one word, break a data format such as JSON, or answer differently on two runs. Each problem has a fix at the moment the next token is picked, with no retraining. Interviewers often ask "Why does my output repeat?" and "How do you make sure the output is valid JSON?"
What you will learn
- Apply repetition, frequency and presence penalties and stop sequences
- Mask logits with a grammar to force valid structure
- Explain sources of nondeterminism even at temperature 0
Key terms
- token
- A word or part of a word that the model picks.
- logit
- The raw score the model gives each possible next token.
- presence penalty
- A flat fee off the score of each used token.
- used token
- A token that is already in the text so far.
Prerequisites
Take Temperature and entropy: how sampling reshapes the distribution first. It comes right before this lesson in the course.
Where this lesson picks up: In the last lesson, you divided the scores by a temperature to make the pick sharper or flatter. At temperature 0 the model always took the top score, but what if the top score goes to the same word again and again?
What comes next
Next, you will see how long each pass through the model takes, and what slows it down.
Next lesson: Prefill vs decode on the roofline
More lessons in this part
- Greedy, beam, top-k and top-p decoding 16 min
- Temperature and entropy: how sampling reshapes the distribution 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.