Multi-Head Attention with Combined QKV Weight Matrix
Problem statement
Implement a causal multi-head self-attention layer that uses a single combined weight matrix to produce queries, keys, and values in one matrix multiplication (instead of three separate projections). Splitting Q/K/V from one fused projection is a common optimization that reduces overhead and improves memory locality.
Given:
x: input tensor of shape(batch, num_tokens, d_in)W_qkv: combined projection matrix of shape(d_in, 3 * d_out)-- the firstd_outoutput columns produce queries, the nextd_outproduce keys, and the lastd_outproduce valuesW_out: output projection matrix of shape(d_out, d_out)num_heads: number of attention heads.d_outis divisible bynum_headsandhead_dim = d_out // num_heads
Your function must:
- Project
xto a combined Q/K/V tensor of shape(batch, num_tokens, 3 * d_out)usingW_qkv(no bias). - Reshape and split it into three separate tensors of shape
(batch, num_heads, num_tokens, head_dim)for queries, keys, and values. - Compute scaled dot-product attention per head with scaling factor
1 / sqrt(head_dim). - Apply a causal mask so position
icannot attend to positionsj > i(use-infbefore softmax). - Apply softmax along the last (key) dimension, multiply by values, then concatenate heads back to shape
(batch, num_tokens, d_out). - Apply the output projection
W_outand return the result as a nested Python list.
No dropout is applied. Use only NumPy and pure Python.
Example
Input:
x = [[[1.0, 0.0], [0.0, 1.0]]] (shape (1,2,2)), W_qkv stacks three copies of I_2 horizontally so Q=K=V=x, W_out = I_2, num_heads = 1
Output:
[[[1.0, 0.0], [0.3302, 0.6698]]]
Reasoning: With the combined projection, queries, keys, and values all equal x. head_dim = 2 so scores = Q K^T / sqrt(2) = [[1, 0], [0, 1]] / sqrt(2). Apply the causal mask: row 0 becomes [0.7071, -inf] which softmaxes to [1, 0]; row 1 is [0, 0.7071] which softmaxes to [0.3302, 0.6698]. Multiplying by V = [[1,0],[0,1]] gives row 0 = [1, 0] and row 1 = 0.3302[1,0] + 0.6698[0,1] = [0.3302, 0.6698]. The identity output projection leaves the result unchanged.
Show hint
How can three projections of the same input be done with a single matrix product?
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium