Debug a Minimal GPT: Mask, Training Step, Positions, KV Cache
Problem statement
How would you debug a small character-level GPT whose training loss stays near chance? It uses right-padded sequences and ignores padding in the loss. The notebook has bugs in attention masking, the training loop and the input representation, while the transformer block itself is correct.
Find the bugs, explain their effects and give fixes with tests.
Show hint
Map each symptom to its class of cause: a loss that never moves, a loss that is suspiciously low, and a loss that plateaus.
Relevant roles: ML Engineer, Research Scientist, LLM Engineer and Deep Learning Engineer.
Related free problems
- Debugging a Misbehaving Transformer Training PipelineDeep LearningHard
- Fix a Buggy Transformer, Then Add a KV CacheML CodingHard