Chapter 3: LLM foundations
Next-token probabilities: cross-entropy and perplexity
A language model learns one skill: it predicts the next token. In this lesson you build its practice questions from plain text. Then you score its guesses with cross-entropy and turn that score into perplexity. Interviewers often ask what perplexity means and how training targets come from text.
What you will learn
- Build shifted input/target pairs for next-token training
- Convert average cross-entropy to perplexity
- Explain why perplexity is not comparable across tokenizers
Key terms
- next token
- the token right after the ones the model has seen
- softmax
- turns scores into probabilities that add to 1
- cross-entropy
- the loss −ln p, where p is the right answer's probability
- perplexity
- one score for how well a model predicts text
- stride
- how many tokens you slide before the next window starts
Prerequisites
Take Subword tokens and byte-pair encoding first. It comes right before this lesson in the course.
Where this lesson picks up: You used byte-pair encoding to turn text into a list of token IDs, such as 40, 367, 2885. A list of numbers alone does not train a model, so what does the model learn from it?
What comes next
Next, you will see how tokenizer choices cause odd model behavior and why new special tokens change the vocabulary.
Next lesson: Tokenizer failure modes, special tokens and vocabulary size
More lessons in this part
- Subword tokens and byte-pair encoding 16 min
- Tokenizer failure modes, special tokens and vocabulary size 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.