Debugging a Misbehaving Transformer Training Pipeline

Reported in interviews at OpenAI Lila

Problem statement

How would you debug a Transformer training pipeline with several independent bugs? An encoder-decoder model has attention-mask errors, poor validation results, mixed-precision crashes and inconsistent runs. You can use a tiny copy task to isolate failures quickly.

Isolate the bugs and explain the correct mixed-precision training sequence.

Show hint

Which of the four symptoms is a crash, which is a generalization gap, which appears only under one precision setting, and which concerns repeatability, and which pipeline stage could produce each?

Relevant roles: ML Engineer, Deep Learning Engineer and Research Scientist.

Related free problems