Transformer Training Choices: Norms, Activations, Positions, Schedules, Precision and Masks
Problem statement
How would you configure and debug a decoder-only language model trained from scratch? The team uses AdamW and data-parallel training. You need to explain how normalization, numeric precision and attention masks affect training.
Explain normalization, precision and causal-mask failures.
Show hint
For each symptom, ask which quantity in the network (gradient scale, unit activity, input distribution, numeric range, or information flow) is behaving abnormally.
Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and Research Scientist.