Debugging a Misbehaving Transformer Training Pipeline
Problem statement
How would you debug a Transformer training pipeline with several independent bugs? An encoder-decoder model has attention-mask errors, poor validation results, mixed-precision crashes and inconsistent runs. You can use a tiny copy task to isolate failures quickly.
Isolate the bugs and explain the correct mixed-precision training sequence.
Show hint
Which of the four symptoms is a crash, which is a generalization gap, which appears only under one precision setting, and which concerns repeatability, and which pipeline stage could produce each?
Relevant roles: ML Engineer, Deep Learning Engineer and Research Scientist.
Related free problems
- Debug a Minimal GPT: Mask, Training Step, Positions, KV CacheLLMs & GenAIHard