Language models

Attention

Imagine asking classmates for help and giving more weight to those who know the topic. Attention lets each token gather information from other tokens with learned weights. Queries and keys choose the weights; values carry the information being mixed.

Attention weights sum to 1: the weighted value is 0.25 × 2 + 0.75 × 6 = 5.

How it works

  1. Project token representations into queries, keys, and values.
  2. Compute query-key dot products and divide by $\sqrt{d_k}$ to control score scale.
  3. Apply any mask before a row-wise softmax turns scores into weights.
  4. Multiply the weights by the values; with multiple heads, concatenate head outputs and project them.

$$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V$$

The knobs that matter

number of heads
allows separate learned attention patterns
head dimension
sets each head’s representation size and dot-product scale
mask
controls which query-key pairs are allowed

What interviewers ask

  • Why do we divide attention scores by the square root of the key dimension?
  • What are the shapes of Q, K, V and the attention matrix?
  • How do you implement a causal mask without leaking future tokens?
  • What does multi-head attention add, and how does sequence length affect cost?

Common mistakes

  • Applying softmax across the wrong axis instead of across keys for each query.
  • Masking scores with zero rather than a sufficiently negative value before softmax.
  • Treating attention weights as a complete explanation of the model’s decision.

When to use it

Use attention when representations need information from other positions or another sequence. Full attention becomes costly for long sequences because its pairwise score count grows quadratically.

Implement it (12 problems)

Learn it

Interview questions about attention