LLM Fundamentals: Attention Cost, Causal Masks and RLHF
Problem statement
How much does attention cost, and what breaks when a language model can see its next token? Calculate a layer's FLOPs, diagnose a causal-mask bug, then explain the role of a KL penalty in feedback training.
Show hint
For each topic, ask what concrete quantity or failure the concept controls: FLOPs, which positions are visible, how far the policy can move.
Relevant roles: ML Engineer, LLM Engineer, AI Engineer and Gen AI Engineer.
Related free problems
- Compute Attention Memory Traffic and FLOPsML CodingMedium