Chapter 3: LLM foundations

Tokenizer failure modes, special tokens and vocabulary size

Chatbots miscount letters, make mistakes on long sums and charge more for some languages. A big part of the cause is the tokenizer, the step before the model. You will see why, estimate a token bill, and add new chat tokens the right way. Interviewers ask about this to check that you know the layer below the model.

16 minInteractive lessonChapter 3: LLM foundationsLesson 3 of 12

What you will learn

  • Explain why digits and rare words tokenize badly
  • Add special tokens and resize the embedding matrix correctly
  • Estimate token counts and cost for different languages

Key terms

token ID
the number the tokenizer gives each token
special token
a reserved token that marks structure, such as a speaker
embedding table
one row of numbers for each token ID
output head
a table that gives each token a score

Prerequisites

Take Next-token probabilities: cross-entropy and perplexity first. It comes right before this lesson in the course.

Where this lesson picks up: You turned a token list into input and target pairs and scored a model with cross-entropy and perplexity. You saw that two tokenizers can cut one sentence into different numbers of tokens, and that this changes the score.

What comes next

Each token is now a row of numbers, so how does the model decide which earlier tokens matter?

Next lesson: Scaled dot-product attention

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.