Multi-Head Causal Self-Attention From Scratch
Problem statement
Implement the forward pass of multi-head causal self-attention, the core of a decoder-only language model, in numpy.
Given X of shape $T \times D$ (one row per token) and projection matrices Wq, Wk, Wv, Wo, each $D \times D$:
- Compute $Q = XW_q$, $K = XW_k$, $V = XW_v$.
- Split each into
n_headsheads of width $d_h = D / n_{heads}$: head $h$ takes columns $[h\,d_h, (h+1)\,d_h)$. - Per head, scores are $Q_h K_h^{\top} / \sqrt{d_h}$. Position $i$ may only attend to positions $j \le i$ (causal mask). Apply a numerically stable softmax over each row's allowed positions and multiply by $V_h$.
- Concatenate the head outputs back in head order ($T \times D$) and multiply by $W_o$.
No biases, no dropout. Return the $T \times D$ output as a list of lists.
def causal_self_attention(X: list, Wq: list, Wk: list, Wv: list, Wo: list, n_heads: int) -> list:
...
Constraints:
- $1 \le T \le 8$,
n_headsin $\{1, 2, 4\}$, $1 \le d_h \le 4$. - Some tests produce scores above 1000, so exponentiating without subtracting the row max overflows.
- numpy is available. Checked to a relative tolerance of $10^{-7}$.
Examples:
Example 1: with $X = I_2$, every weight $= I_2$ and 1 head -> [[1.0, 0.0], [0.3302, 0.6698]]. Token 0 sees only itself. Token 1 has scores $(0, 1)/\sqrt{2}$, giving weights $(0.3302, 0.6698)$ on $V = I_2$.
Example 2: X = [[0.3, -0.7, 1.2]], Wq = Wk = Wo = I_3, Wv = 2 I_3, 3 heads -> [[0.6, -1.4, 2.4]]. With a single token every head attends to itself with weight 1, so the output is $X W_v W_o$.
Example 3: X = [[40, 0], [0, 40], [40, 40]] with identity weights and 1 head -> [[40, 0], [0, 40], [40, 40]]. The self-scores are about 1131 and 2263, so each token puts essentially all weight on itself; an unshifted exp returns inf.
Show hint
How do you let several heads share one projection while each works on its own slice of the representation?
Relevant roles: LLM Engineer, Deep Learning Engineer, ML Engineer and Research Scientist.
Related free problems
- Masked Self-AttentionMedium
- Multi-Head AttentionMedium