Transformer Attention: Scaling, Heads and Memory
Problem statement
Why scale attention scores, and what changes when you split attention into multiple heads? Consider a decoder-only transformer with model width 4096 and 32 heads of width 128. Then calculate the memory needed to store its attention scores.
Show hint
For each component, ask what would go wrong in training or inference if it were removed.
Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Research Scientist.