Regularizers, Adam vs AdamW and the Decoder Block

Reported in interviews at Snap

Problem statement

How do regularization, optimizers and attention work in practice? The team has a sparse logistic click model, Model S, and a decoder transformer for a writing assistant, Model T. You need to explain why their training choices behave differently.

Explain regularization, optimizer choices and the equations of attention and a decoder block, including causal masking.

Show hint

For each technique, ask what happens to a single weight or a single attention row, rather than to the model as a whole.

Relevant roles: ML Engineer, AI Engineer, Research Scientist and LLM Engineer.

Related free problems