Chapter 3: LLM foundations

Next-token probabilities: cross-entropy and perplexity

A language model learns one skill: it predicts the next token. In this lesson you build its practice questions from plain text. Then you score its guesses with cross-entropy and turn that score into perplexity. Interviewers often ask what perplexity means and how training targets come from text.

17 minInteractive lessonChapter 3: LLM foundationsLesson 2 of 12

What you will learn

  • Build shifted input/target pairs for next-token training
  • Convert average cross-entropy to perplexity
  • Explain why perplexity is not comparable across tokenizers

Key terms

next token
the token right after the ones the model has seen
softmax
turns scores into probabilities that add to 1
cross-entropy
the loss −ln p, where p is the right answer's probability
perplexity
one score for how well a model predicts text
stride
how many tokens you slide before the next window starts

Prerequisites

Take Subword tokens and byte-pair encoding first. It comes right before this lesson in the course.

Where this lesson picks up: You used byte-pair encoding to turn text into a list of token IDs, such as 40, 367, 2885. A list of numbers alone does not train a model, so what does the model learn from it?

What comes next

Next, you will see how tokenizer choices cause odd model behavior and why new special tokens change the vocabulary.

Next lesson: Tokenizer failure modes, special tokens and vocabulary size

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.