Multi-Query Attention (MQA)

MediumML Coding

Problem statement

Implement Multi-Query Attention (MQA), an efficient variant of multi-head attention. In standard multi-head attention every head has its own query, key and value projections. In MQA each head keeps its own query projection, but all heads share a single key projection and a single value projection. That shrinks the key-value cache by a factor equal to the number of heads, which cuts memory bandwidth during autoregressive decoding.

Write multiquery_attention(X, W_queries, W_key, W_value, W_out) where:

  • X: input of shape (seq_len, d_model)
  • W_queries: a list of arrays, each (d_model, d_k), one per head
  • W_key: one (d_model, d_k) array shared by all heads
  • W_value: one (d_model, d_v) array shared by all heads
  • W_out: the output projection, shape (num_heads * d_v, d_model)

For each head $h$: $Q_h = XW_h^Q$, $K = XW^K$, $V = XW^V$, and $\text{head}_h = \text{softmax}(Q_h K^\top / \sqrt{d_k})\, V$ with a numerically stable row-wise softmax. Concatenate the heads in order along the feature axis, multiply by W_out, and return the (seq_len, d_model) result rounded to 4 decimals.

Example

Input:

X = np.array([[1.0, 0.0], [0.0, 1.0]])
W_queries = [np.eye(2)]
W_key = np.eye(2)
W_value = np.eye(2)
W_out = np.eye(2)
multiquery_attention(X, W_queries, W_key, W_value, W_out)

Output:

[[0.6698, 0.3302], [0.3302, 0.6698]]

Reasoning: with one head and identity projections, $Q = K = V = X$. The scores are $XX^\top / \sqrt{2} = I / \sqrt{2}$, so row 0 is [0.7071, 0], whose softmax is [0.6698, 0.3302]. Multiplying by $V$ and the identity output projection gives the result above.

Show hint

Which projections in multi-head attention would you share across heads if your bottleneck were the memory used by cached keys and values?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, AI Engineer and Gen AI Engineer.

Related free problems