Language models
Attention
Imagine asking classmates for help and giving more weight to those who know the topic. Attention lets each token gather information from other tokens with learned weights. Queries and keys choose the weights; values carry the information being mixed.
| value 2 | value 6 | |
|---|---|---|
| query token | 0.25 | 0.75 |
How it works
- Project token representations into queries, keys, and values.
- Compute query-key dot products and divide by $\sqrt{d_k}$ to control score scale.
- Apply any mask before a row-wise softmax turns scores into weights.
- Multiply the weights by the values; with multiple heads, concatenate head outputs and project them.
$$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V$$
The knobs that matter
- number of heads
- allows separate learned attention patterns
- head dimension
- sets each head’s representation size and dot-product scale
- mask
- controls which query-key pairs are allowed
What interviewers ask
- Why do we divide attention scores by the square root of the key dimension?
- What are the shapes of Q, K, V and the attention matrix?
- How do you implement a causal mask without leaking future tokens?
- What does multi-head attention add, and how does sequence length affect cost?
Common mistakes
- Applying softmax across the wrong axis instead of across keys for each query.
- Masking scores with zero rather than a sufficiently negative value before softmax.
- Treating attention weights as a complete explanation of the model’s decision.
When to use it
Use attention when representations need information from other positions or another sequence. Full attention becomes costly for long sequences because its pairwise score count grows quadratically.
Implement it (12 problems)
- Multi-Head AttentionFreeMediumSolve
- Masked Self-AttentionFreeMediumSolve
- Scaled Dot-Product AttentionFreeMediumSolve
- Batched Causal Self-AttentionFreeMediumSolve
- Gated AttentionFreeMediumSolve