Transformer Training Choices: Norms, Activations, Positions, Schedules, Precision and Masks

MediumDeep Learning

Reported in interviews at DRW

Problem statement

How would you configure and debug a decoder-only language model trained from scratch? The team uses AdamW and data-parallel training. You need to explain how normalization, numeric precision and attention masks affect training.

Explain normalization, precision and causal-mask failures.

Show hint

For each symptom, ask which quantity in the network (gradient scale, unit activity, input distribution, numeric range, or information flow) is behaving abnormally.

Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and Research Scientist.

Related free problems