Debug a Minimal GPT: Mask, Training Step, Positions, KV Cache

HardLLMs & GenAI

Reported in interviews at Applied Intuition OpenAI Mercor

Problem statement

How would you debug a small character-level GPT whose training loss stays near chance? It uses right-padded sequences and ignores padding in the loss. The notebook has bugs in attention masking, the training loop and the input representation, while the transformer block itself is correct.

Find the bugs, explain their effects and give fixes with tests.

Show hint

Map each symptom to its class of cause: a loss that never moves, a loss that is suspiciously low, and a loss that plateaus.

Relevant roles: ML Engineer, Research Scientist, LLM Engineer and Deep Learning Engineer.

Related free problems