Scaled Dot-Product Attention
Problem statement
Implement scaled dot-product attention, the core building block of Transformer architectures. Given precomputed query, key and value matrices, compute the attention output and the attention weight matrix.
Your function should:
- Compute attention scores from
QandKwith the appropriate scaling. - Optionally apply a mask before computing the weights (mask value 0 blocks a position, 1 keeps it).
- Apply a numerically stable softmax to get the attention weights.
- Compute the weighted sum of the values.
Write scaled_dot_product_attention(Q, K, V, mask=None) returning a tuple (output, attention_weights).
Inputs:
Q: shape(seq_len_q, d_k).K: shape(seq_len_k, d_k).V: shape(seq_len_k, d_v).mask: optional binary array of shape(seq_len_q, seq_len_k); 0 = block, 1 = attend.
Outputs:
outputwith shape(seq_len_q, d_v).attention_weightswith shape(seq_len_q, seq_len_k).
Example
Input:
Q = [[1.0, 0.0], [0.0, 1.0]]
K = [[1.0, 0.0], [0.0, 1.0]]
V = [[1.0, 2.0], [3.0, 4.0]]
Output:
output = [[1.6605, 2.6605], [2.3395, 3.3395]]
weights = [[0.6698, 0.3302], [0.3302, 0.6698]]
Reasoning: with $d_k = 2$ the scores are $QK^T/\sqrt{2} = [[0.7071, 0], [0, 0.7071]]$. The softmax of row 0 is $[0.6698, 0.3302]$, so the first output row is $0.6698 \cdot [1, 2] + 0.3302 \cdot [3, 4] = [1.6605, 2.6605]$.
Show hint
How do you turn similarity scores between each query and every key into a distribution over keys?
Relevant roles: ML Engineer, LLM Engineer, AI Engineer, Deep Learning Engineer and Research Scientist.
Related free problems
- Gated AttentionML CodingMedium
- Scale Attention Scores by sqrt(d_k)ML CodingEasy
- Multi-Query Attention (MQA)ML CodingMedium
- Multi-Head AttentionML CodingMedium