Transformer Attention and the 1/sqrt(d_k) Scaling
Problem statement
How does scaled dot-product attention combine information across a sequence? It uses queries, keys, and values, and scales query-key scores before applying softmax. Explain the shapes and lookup, derive the scaling, and compare the cost and behavior of multiple heads.
Show hint
Which property of the logits entering the softmax depends on the head width?
Relevant roles: LLM Engineer, Deep Learning Engineer, Research Scientist and ML Engineer.
Related free problems
- Scale Attention Scores by sqrt(d_k)ML CodingEasy