Regularizers, Adam vs AdamW and the Decoder Block
Problem statement
How do regularization, optimizers and attention work in practice? The team has a sparse logistic click model, Model S, and a decoder transformer for a writing assistant, Model T. You need to explain why their training choices behave differently.
Explain regularization, optimizer choices and the equations of attention and a decoder block, including causal masking.
Show hint
For each technique, ask what happens to a single weight or a single attention row, rather than to the model as a whole.
Relevant roles: ML Engineer, AI Engineer, Research Scientist and LLM Engineer.
Related free problems
- AdamW Optimizer StepML CodingEasy