Chapter 4: LLM inference

Repetition, penalties, constrained decoding and determinism

Models sometimes repeat one word, break a data format such as JSON, or answer differently on two runs. Each problem has a fix at the moment the next token is picked, with no retraining. Interviewers often ask "Why does my output repeat?" and "How do you make sure the output is valid JSON?"

18 minInteractive lessonChapter 4: LLM inferenceLesson 3 of 12

What you will learn

  • Apply repetition, frequency and presence penalties and stop sequences
  • Mask logits with a grammar to force valid structure
  • Explain sources of nondeterminism even at temperature 0

Key terms

token
A word or part of a word that the model picks.
logit
The raw score the model gives each possible next token.
presence penalty
A flat fee off the score of each used token.
used token
A token that is already in the text so far.

Prerequisites

Take Temperature and entropy: how sampling reshapes the distribution first. It comes right before this lesson in the course.

Where this lesson picks up: In the last lesson, you divided the scores by a temperature to make the pick sharper or flatter. At temperature 0 the model always took the top score, but what if the top score goes to the same word again and again?

What comes next

Next, you will see how long each pass through the model takes, and what slows it down.

Next lesson: Prefill vs decode on the roofline

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.