Language models
Tokenizer
Like cutting a long word into reusable puzzle pieces, a tokenizer splits text into units a model can read. It maps those pieces to integer IDs. Subword tokenizers reuse common pieces instead of needing a separate entry for every possible word.
How it works
- Choose normalization, initial symbols, and rules for preserving spaces and special tokens.
- For byte-pair encoding, count adjacent symbol pairs in the training text and repeatedly merge the most frequent pair.
- Encode new text using the learned merge priorities and map the resulting pieces to IDs.
- Decode IDs back to text using the same vocabulary and decoding rules.
The knobs that matter
- vocabulary size
- larger vocabularies often shorten sequences but enlarge embedding tables
- normalization
- changes which text distinctions survive encoding
- byte fallback
- represents otherwise unknown text as bytes
What interviewers ask
- How would you train and apply a byte-pair encoding tokenizer?
- Why might one language or a string of numbers use many more tokens than another?
- What tokenizer mismatches would you check if a model suddenly produced poor output?
Common mistakes
- Assuming one token always means one word.
- Changing a trained model’s token-to-ID mapping without updating its embeddings and output head.
- Assuming every tokenizer reproduces the original text exactly after normalization.
When to use it
Use the tokenizer paired with the trained model. Train a new tokenizer when building a model with text or language needs that the existing vocabulary handles poorly.
Implement it (6 problems)
Learn it
Interview questions about tokenizer
- Tokenizers and the Modern LLM BlockFreeMediumSolve