Chapter 3: LLM foundations

Subword tokens and byte-pair encoding

A language model cannot read letters. It reads numbers, one for each small piece of text. The way you cut text into pieces changes how long the text gets and which words the model can write. Interviewers often ask you to build byte-pair encoding, a method that many large models use, and to explain why it cuts words into parts.

16 minInteractive lessonChapter 3: LLM foundationsLesson 1 of 12

What you will learn

  • Learn BPE merges from a toy corpus
  • Encode new text with a merge table
  • Explain the vocabulary-size versus sequence-length trade-off

Key terms

token
one piece of text that the model reads
vocabulary
the fixed list of tokens that a model knows
merge
one step that glues two side-by-side tokens into one

Prerequisites

Take VAE vs GAN vs diffusion: quality, diversity, cost and FID first. It comes right before this lesson in the course.

Where this lesson picks up: none (this is the first lesson of the chapter).

What comes next

Each piece of text is now a token from a fixed list, so how does a model score its guess for the next token?

Next lesson: Next-token probabilities: cross-entropy and perplexity

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.