Multi-Head Causal Self-Attention From Scratch

HardML Coding

Reported in interviews at Amazon Anthropic Apple Bytedance Cohere Google

Problem statement

Implement the forward pass of multi-head causal self-attention, the core of a decoder-only language model, in numpy.

Given X of shape $T \times D$ (one row per token) and projection matrices Wq, Wk, Wv, Wo, each $D \times D$:

  1. Compute $Q = XW_q$, $K = XW_k$, $V = XW_v$.
  2. Split each into n_heads heads of width $d_h = D / n_{heads}$: head $h$ takes columns $[h\,d_h, (h+1)\,d_h)$.
  3. Per head, scores are $Q_h K_h^{\top} / \sqrt{d_h}$. Position $i$ may only attend to positions $j \le i$ (causal mask). Apply a numerically stable softmax over each row's allowed positions and multiply by $V_h$.
  4. Concatenate the head outputs back in head order ($T \times D$) and multiply by $W_o$.

No biases, no dropout. Return the $T \times D$ output as a list of lists.

def causal_self_attention(X: list, Wq: list, Wk: list, Wv: list, Wo: list, n_heads: int) -> list:
    ...

Constraints:

  • $1 \le T \le 8$, n_heads in $\{1, 2, 4\}$, $1 \le d_h \le 4$.
  • Some tests produce scores above 1000, so exponentiating without subtracting the row max overflows.
  • numpy is available. Checked to a relative tolerance of $10^{-7}$.

Examples:

Example 1: with $X = I_2$, every weight $= I_2$ and 1 head -> [[1.0, 0.0], [0.3302, 0.6698]]. Token 0 sees only itself. Token 1 has scores $(0, 1)/\sqrt{2}$, giving weights $(0.3302, 0.6698)$ on $V = I_2$.

Example 2: X = [[0.3, -0.7, 1.2]], Wq = Wk = Wo = I_3, Wv = 2 I_3, 3 heads -> [[0.6, -1.4, 2.4]]. With a single token every head attends to itself with weight 1, so the output is $X W_v W_o$.

Example 3: X = [[40, 0], [0, 40], [40, 40]] with identity weights and 1 head -> [[40, 0], [0, 40], [40, 40]]. The self-scores are about 1131 and 2263, so each token puts essentially all weight on itself; an unshifted exp returns inf.

Show hint

How do you let several heads share one projection while each works on its own slice of the representation?

Relevant roles: LLM Engineer, Deep Learning Engineer, ML Engineer and Research Scientist.

Related free problems