Transformer Attention and the 1/sqrt(d_k) Scaling

MediumLLMs & GenAI

Reported in interviews at DRW Microsoft TikTok Zoox Swiggy Amazon

Problem statement

How does scaled dot-product attention combine information across a sequence? It uses queries, keys, and values, and scales query-key scores before applying softmax. Explain the shapes and lookup, derive the scaling, and compare the cost and behavior of multiple heads.

Show hint

Which property of the logits entering the softmax depends on the head width?

Relevant roles: LLM Engineer, Deep Learning Engineer, Research Scientist and ML Engineer.

Related free problems