Multi-Head Attention
Problem statement
Implement Q/K/V projections, stable attention, and feature splitting in NumPy. No batching, masks, or output projection.
Interview focus: explain the shapes, why scaling uses the head width, and why softmax runs over keys.
Each head combines token information using its own feature block. T counts tokens; D is the full feature width.
import numpy as np
def compute_qkv(X, W_q, W_k, W_v):
# (T,D), three (D,D) weights -> tuple of three (T,D) arrays.
pass
def self_attention(Q, K, V):
# Three (T,d) arrays -> weighted values (T,d).
pass
def multi_head_attention(Q, K, V, n_heads):
# Three (T,D) arrays -> concatenated heads (T,D).
pass
compute_qkv:Xis(T,D); weights are(D,D). Return(X @ W_q, X @ W_k, X @ W_v), each(T,D).self_attention:Q,K,Vare(T,d). Return(T,d):
scores = Q @ K.T / sqrt(d)
weights = softmax(scores, axis=1)
return weights @ V
Subtract each score row's maximum before exponentiating. Normalize over keys.
multi_head_attention: split Q, K, V into matching contiguous feature blocks,d = D // n_heads. Apply attention independently; concatenate in head order to(T,D). Keep every token row.
{"kind":"boxes","label":"Split feature columns into two heads, then join their outputs","cols":2,"rows":4,"nodes":[{"id":"in","label":"Q, K, V","sub":"each (3,4)","col":0.5,"row":0},{"id":"h1","label":"Q₁, K₁, V₁","sub":"cols 0–1: (3,2)","col":0,"row":1,"tone":"accent"},{"id":"h2","label":"Q₂, K₂, V₂","sub":"cols 2–3: (3,2)","col":1,"row":1,"tone":"teal"},{"id":"o1","label":"O₁ (3,2)","sub":"attention","col":0,"row":2,"tone":"accent"},{"id":"o2","label":"O₂ (3,2)","sub":"attention","col":1,"row":2,"tone":"teal"},{"id":"out","label":"O (3,4)","sub":"output cells oᵢⱼ","col":0.5,"row":3}],"edges":[{"from":"in","to":"h1"},{"from":"in","to":"h2"},{"from":"h1","to":"o1"},{"from":"h2","to":"o2"},{"from":"o1","to":"out"},{"from":"o2","to":"out"}],"caption":"Keep all three token rows. Concatenate O₁ then O₂ along columns."}
n_heads always divides D. Do not call a built-in attention function.
Weights become mixtures
For this illustration, head 1 uses Q₁ = K₁ = np.eye(2). Head 2 uses Q₂ = np.zeros((2,2)), K₂ = np.eye(2).
{"kind":"compare","cols":["Key 1","Key 2"],"rows":[["Head 1 · Query 1","<span style=\"display:block;padding:6px;background:#8b1e2d;color:white\">0.6698</span>","<span style=\"display:block;padding:6px;background:#f6e6e3;color:#1c1915\">0.3302</span>"],["Head 1 · Query 2","<span style=\"display:block;padding:6px;background:#f6e6e3;color:#1c1915\">0.3302</span>","<span style=\"display:block;padding:6px;background:#8b1e2d;color:white\">0.6698</span>"],["Head 2 · Query 1","<span style=\"display:block;padding:6px;background:#2a6670;color:white\">0.5000</span>","<span style=\"display:block;padding:6px;background:#2a6670;color:white\">0.5000</span>"],["Head 2 · Query 2","<span style=\"display:block;padding:6px;background:#2a6670;color:white\">0.5000</span>","<span style=\"display:block;padding:6px;background:#2a6670;color:white\">0.5000</span>"]],"caption":"Each head computes its own weights. Each row sums to 1."}
{"kind":"boxes","label":"Query 1 weights multiply value rows and add","cols":2,"rows":3,"nodes":[{"id":"w","label":"Query 1","sub":"[0.6698, 0.3302]","col":0.5,"row":0},{"id":"v1","label":"0.6698 × [2,0]","col":0,"row":1,"tone":"accent"},{"id":"v2","label":"0.3302 × [0,4]","col":1,"row":1,"tone":"teal"},{"id":"o","label":"Add → output","sub":"[1.3395, 1.3210]","col":0.5,"row":2}],"edges":[{"from":"w","to":"v1"},{"from":"w","to":"v2"},{"from":"v1","to":"o"},{"from":"v2","to":"o"}],"caption":"Multiply using unrounded weights; round only the output."}
Changing V keeps the weights unchanged.
The first test case uses two heads with different values, so you can see each head's output in its own columns.
Show hint
First project X with each weight matrix. Each result keeps T token rows and D feature columns.
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, Gen AI Engineer and Research Scientist.
Related free problems
- Gated AttentionMedium
- Masked Self-AttentionMedium