Tokenizers and the Modern LLM Block

MediumLLMs & GenAI

Reported in interviews at Netflix

Problem statement

How can a tokenizer preserve a symbol it has never seen? Start with unknown-token handling, then count the weights in a decoder block and compare its design choices.

Show hint

For the tokenizer parts, ask what an encoder can tolerate that a text generator cannot.

Relevant roles: ML Engineer, LLM Engineer, AI Engineer and Research Scientist.

Related free problems