Language models

Tokenizer

Like cutting a long word into reusable puzzle pieces, a tokenizer splits text into units a model can read. It maps those pieces to integer IDs. Subword tokenizers reuse common pieces instead of needing a separate entry for every possible word.

Startl o wMerge l + olo wMerge lo + wlow
Illustrative learned merge order: three starting symbols become one token.
Startl o wMerge l + olo wMerge lo + wlow
Illustrative learned merge order: three starting symbols become one token.

How it works

  1. Choose normalization, initial symbols, and rules for preserving spaces and special tokens.
  2. For byte-pair encoding, count adjacent symbol pairs in the training text and repeatedly merge the most frequent pair.
  3. Encode new text using the learned merge priorities and map the resulting pieces to IDs.
  4. Decode IDs back to text using the same vocabulary and decoding rules.

The knobs that matter

vocabulary size
larger vocabularies often shorten sequences but enlarge embedding tables
normalization
changes which text distinctions survive encoding
byte fallback
represents otherwise unknown text as bytes

What interviewers ask

  • How would you train and apply a byte-pair encoding tokenizer?
  • Why might one language or a string of numbers use many more tokens than another?
  • What tokenizer mismatches would you check if a model suddenly produced poor output?

Common mistakes

  • Assuming one token always means one word.
  • Changing a trained model’s token-to-ID mapping without updating its embeddings and output head.
  • Assuming every tokenizer reproduces the original text exactly after normalization.

When to use it

Use the tokenizer paired with the trained model. Train a new tokenizer when building a model with text or language needs that the existing vocabulary handles poorly.

Implement it (6 problems)

Learn it

Interview questions about tokenizer