Scale Attention Scores by sqrt(d_k)
Problem statement
Turn query-key dot products into attention weights. Divide each score by sqrt(d_k), then apply softmax across the keys for each query.
Implement a function scaled_attention_weights(Q, K) that:
- Computes raw attention scores as
Q @ K.T. - Divides the scores by
sqrt(d_k), whered_kis the last dimension ofK. - Applies a numerically stable row-wise softmax to obtain attention weights.
Args:
Q: numpy array of shape(n_q, d_k)K: numpy array of shape(n_k, d_k)
Return the attention weight matrix of shape (n_q, n_k) as a nested Python list, with each value rounded to 4 decimal places.
Example
Input:
Q = [[1.0, 0.0]]
K = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
Output:
[[0.4011, 0.1978, 0.4011]]
Reasoning: Raw scores Q @ K.T = [[1, 0, 1]]. Dividing by sqrt(2) gives [[0.7071, 0, 0.7071]]. A row-wise softmax gives exp(0.7071) / (2 exp(0.7071) + 1) = 0.4011 for the first and third keys and 1 / (2 exp(0.7071) + 1) = 0.1978 for the second.
Show hint
What happens to the spread of a dot product between two random vectors as their length grows?
Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, AI Engineer and Research Scientist.
Related free problems
- Gated AttentionMedium
- Scaled Dot-Product AttentionLLMs & GenAIMedium
- Multi-Head AttentionMedium