Multi-Head Attention

MediumML Coding

Reported in interviews at Google Uber

Problem statement

Implement Q/K/V projections, stable attention, and feature splitting in NumPy. No batching, masks, or output projection.

Interview focus: explain the shapes, why scaling uses the head width, and why softmax runs over keys.

Each head combines token information using its own feature block. T counts tokens; D is the full feature width.

import numpy as np

def compute_qkv(X, W_q, W_k, W_v):
    # (T,D), three (D,D) weights -> tuple of three (T,D) arrays.
    pass

def self_attention(Q, K, V):
    # Three (T,d) arrays -> weighted values (T,d).
    pass

def multi_head_attention(Q, K, V, n_heads):
    # Three (T,D) arrays -> concatenated heads (T,D).
    pass
  • compute_qkv: X is (T,D); weights are (D,D). Return (X @ W_q, X @ W_k, X @ W_v), each (T,D).
  • self_attention: Q, K, V are (T,d). Return (T,d):
scores = Q @ K.T / sqrt(d)
weights = softmax(scores, axis=1)
return weights @ V

Subtract each score row's maximum before exponentiating. Normalize over keys.

  • multi_head_attention: split Q, K, V into matching contiguous feature blocks, d = D // n_heads. Apply attention independently; concatenate in head order to (T,D). Keep every token row.
{"kind":"boxes","label":"Split feature columns into two heads, then join their outputs","cols":2,"rows":4,"nodes":[{"id":"in","label":"Q, K, V","sub":"each (3,4)","col":0.5,"row":0},{"id":"h1","label":"Q₁, K₁, V₁","sub":"cols 0–1: (3,2)","col":0,"row":1,"tone":"accent"},{"id":"h2","label":"Q₂, K₂, V₂","sub":"cols 2–3: (3,2)","col":1,"row":1,"tone":"teal"},{"id":"o1","label":"O₁ (3,2)","sub":"attention","col":0,"row":2,"tone":"accent"},{"id":"o2","label":"O₂ (3,2)","sub":"attention","col":1,"row":2,"tone":"teal"},{"id":"out","label":"O (3,4)","sub":"output cells oᵢⱼ","col":0.5,"row":3}],"edges":[{"from":"in","to":"h1"},{"from":"in","to":"h2"},{"from":"h1","to":"o1"},{"from":"h2","to":"o2"},{"from":"o1","to":"out"},{"from":"o2","to":"out"}],"caption":"Keep all three token rows. Concatenate O₁ then O₂ along columns."}

n_heads always divides D. Do not call a built-in attention function.

Weights become mixtures

For this illustration, head 1 uses Q₁ = K₁ = np.eye(2). Head 2 uses Q₂ = np.zeros((2,2)), K₂ = np.eye(2).

{"kind":"compare","cols":["Key 1","Key 2"],"rows":[["Head 1 · Query 1","<span style=\"display:block;padding:6px;background:#8b1e2d;color:white\">0.6698</span>","<span style=\"display:block;padding:6px;background:#f6e6e3;color:#1c1915\">0.3302</span>"],["Head 1 · Query 2","<span style=\"display:block;padding:6px;background:#f6e6e3;color:#1c1915\">0.3302</span>","<span style=\"display:block;padding:6px;background:#8b1e2d;color:white\">0.6698</span>"],["Head 2 · Query 1","<span style=\"display:block;padding:6px;background:#2a6670;color:white\">0.5000</span>","<span style=\"display:block;padding:6px;background:#2a6670;color:white\">0.5000</span>"],["Head 2 · Query 2","<span style=\"display:block;padding:6px;background:#2a6670;color:white\">0.5000</span>","<span style=\"display:block;padding:6px;background:#2a6670;color:white\">0.5000</span>"]],"caption":"Each head computes its own weights. Each row sums to 1."}
{"kind":"boxes","label":"Query 1 weights multiply value rows and add","cols":2,"rows":3,"nodes":[{"id":"w","label":"Query 1","sub":"[0.6698, 0.3302]","col":0.5,"row":0},{"id":"v1","label":"0.6698 × [2,0]","col":0,"row":1,"tone":"accent"},{"id":"v2","label":"0.3302 × [0,4]","col":1,"row":1,"tone":"teal"},{"id":"o","label":"Add → output","sub":"[1.3395, 1.3210]","col":0.5,"row":2}],"edges":[{"from":"w","to":"v1"},{"from":"w","to":"v2"},{"from":"v1","to":"o"},{"from":"v2","to":"o"}],"caption":"Multiply using unrounded weights; round only the output."}

Changing V keeps the weights unchanged.

The first test case uses two heads with different values, so you can see each head's output in its own columns.

Show hint

First project X with each weight matrix. Each result keeps T token rows and D feature columns.

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, Gen AI Engineer and Research Scientist.

Related free problems