Chapter 3: LLM foundations
Tokenizer failure modes, special tokens and vocabulary size
Chatbots miscount letters, make mistakes on long sums and charge more for some languages. A big part of the cause is the tokenizer, the step before the model. You will see why, estimate a token bill, and add new chat tokens the right way. Interviewers ask about this to check that you know the layer below the model.
What you will learn
- Explain why digits and rare words tokenize badly
- Add special tokens and resize the embedding matrix correctly
- Estimate token counts and cost for different languages
Key terms
- token ID
- the number the tokenizer gives each token
- special token
- a reserved token that marks structure, such as a speaker
- embedding table
- one row of numbers for each token ID
- output head
- a table that gives each token a score
Prerequisites
Take Next-token probabilities: cross-entropy and perplexity first. It comes right before this lesson in the course.
Where this lesson picks up: You turned a token list into input and target pairs and scored a model with cross-entropy and perplexity. You saw that two tokenizers can cut one sentence into different numbers of tokens, and that this changes the score.
What comes next
Each token is now a row of numbers, so how does the model decide which earlier tokens matter?
Next lesson: Scaled dot-product attention
More lessons in this part
- Subword tokens and byte-pair encoding 16 min
- Next-token probabilities: cross-entropy and perplexity 17 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.