Multi-Head Attention with Combined QKV Weight Matrix

HardML Coding

Problem statement

Implement a causal multi-head self-attention layer that uses a single combined weight matrix to produce queries, keys, and values in one matrix multiplication (instead of three separate projections). Splitting Q/K/V from one fused projection is a common optimization that reduces overhead and improves memory locality.

Given:

  • x: input tensor of shape (batch, num_tokens, d_in)
  • W_qkv: combined projection matrix of shape (d_in, 3 * d_out) -- the first d_out output columns produce queries, the next d_out produce keys, and the last d_out produce values
  • W_out: output projection matrix of shape (d_out, d_out)
  • num_heads: number of attention heads. d_out is divisible by num_heads and head_dim = d_out // num_heads

Your function must:

  1. Project x to a combined Q/K/V tensor of shape (batch, num_tokens, 3 * d_out) using W_qkv (no bias).
  2. Reshape and split it into three separate tensors of shape (batch, num_heads, num_tokens, head_dim) for queries, keys, and values.
  3. Compute scaled dot-product attention per head with scaling factor 1 / sqrt(head_dim).
  4. Apply a causal mask so position i cannot attend to positions j > i (use -inf before softmax).
  5. Apply softmax along the last (key) dimension, multiply by values, then concatenate heads back to shape (batch, num_tokens, d_out).
  6. Apply the output projection W_out and return the result as a nested Python list.

No dropout is applied. Use only NumPy and pure Python.

Example

Input:

x = [[[1.0, 0.0], [0.0, 1.0]]] (shape (1,2,2)), W_qkv stacks three copies of I_2 horizontally so Q=K=V=x, W_out = I_2, num_heads = 1

Output:

[[[1.0, 0.0], [0.3302, 0.6698]]]

Reasoning: With the combined projection, queries, keys, and values all equal x. head_dim = 2 so scores = Q K^T / sqrt(2) = [[1, 0], [0, 1]] / sqrt(2). Apply the causal mask: row 0 becomes [0.7071, -inf] which softmaxes to [1, 0]; row 1 is [0, 0.7071] which softmaxes to [0.3302, 0.6698]. Multiplying by V = [[1,0],[0,1]] gives row 0 = [1, 0] and row 1 = 0.3302[1,0] + 0.6698[0,1] = [0.3302, 0.6698]. The identity output projection leaves the result unchanged.

Show hint

How can three projections of the same input be done with a single matrix product?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.

Related free problems