Transformer Attention: Scaling, Heads and Memory

MediumLLMs & GenAI

Reported in interviews at UiPath

Problem statement

Why scale attention scores, and what changes when you split attention into multiple heads? Consider a decoder-only transformer with model width 4096 and 32 heads of width 128. Then calculate the memory needed to store its attention scores.

Show hint

For each component, ask what would go wrong in training or inference if it were removed.

Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Research Scientist.

Related free problems