Multi-Query Attention (MQA)
Problem statement
Implement Multi-Query Attention (MQA), an efficient variant of multi-head attention. In standard multi-head attention every head has its own query, key and value projections. In MQA each head keeps its own query projection, but all heads share a single key projection and a single value projection. That shrinks the key-value cache by a factor equal to the number of heads, which cuts memory bandwidth during autoregressive decoding.
Write multiquery_attention(X, W_queries, W_key, W_value, W_out) where:
X: input of shape(seq_len, d_model)W_queries: a list of arrays, each(d_model, d_k), one per headW_key: one(d_model, d_k)array shared by all headsW_value: one(d_model, d_v)array shared by all headsW_out: the output projection, shape(num_heads * d_v, d_model)
For each head $h$: $Q_h = XW_h^Q$, $K = XW^K$, $V = XW^V$, and $\text{head}_h = \text{softmax}(Q_h K^\top / \sqrt{d_k})\, V$ with a numerically stable row-wise softmax. Concatenate the heads in order along the feature axis, multiply by W_out, and return the (seq_len, d_model) result rounded to 4 decimals.
Example
Input:
X = np.array([[1.0, 0.0], [0.0, 1.0]])
W_queries = [np.eye(2)]
W_key = np.eye(2)
W_value = np.eye(2)
W_out = np.eye(2)
multiquery_attention(X, W_queries, W_key, W_value, W_out)
Output:
[[0.6698, 0.3302], [0.3302, 0.6698]]
Reasoning: with one head and identity projections, $Q = K = V = X$. The scores are $XX^\top / \sqrt{2} = I / \sqrt{2}$, so row 0 is [0.7071, 0], whose softmax is [0.6698, 0.3302]. Multiplying by $V$ and the identity output projection gives the result above.
Show hint
Which projections in multi-head attention would you share across heads if your bottleneck were the memory used by cached keys and values?
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, AI Engineer and Gen AI Engineer.
Related free problems
- Gated AttentionMedium
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium