Tokenizers and the Modern LLM Block
Problem statement
How can a tokenizer preserve a symbol it has never seen? Start with unknown-token handling, then count the weights in a decoder block and compare its design choices.
Show hint
For the tokenizer parts, ask what an encoder can tolerate that a text generator cannot.
Relevant roles: ML Engineer, LLM Engineer, AI Engineer and Research Scientist.
Related free problems
- Grammar-Constrained Token MaskingML CodingMedium