Scale Attention Scores by sqrt(d_k)

EasyML Coding

Problem statement

Turn query-key dot products into attention weights. Divide each score by sqrt(d_k), then apply softmax across the keys for each query.

Implement a function scaled_attention_weights(Q, K) that:

  1. Computes raw attention scores as Q @ K.T.
  2. Divides the scores by sqrt(d_k), where d_k is the last dimension of K.
  3. Applies a numerically stable row-wise softmax to obtain attention weights.

Args:

  • Q: numpy array of shape (n_q, d_k)
  • K: numpy array of shape (n_k, d_k)

Return the attention weight matrix of shape (n_q, n_k) as a nested Python list, with each value rounded to 4 decimal places.

Example

Input:

Q = [[1.0, 0.0]]
K = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]

Output:

[[0.4011, 0.1978, 0.4011]]

Reasoning: Raw scores Q @ K.T = [[1, 0, 1]]. Dividing by sqrt(2) gives [[0.7071, 0, 0.7071]]. A row-wise softmax gives exp(0.7071) / (2 exp(0.7071) + 1) = 0.4011 for the first and third keys and 1 / (2 exp(0.7071) + 1) = 0.1978 for the second.

Show hint

What happens to the spread of a dot product between two random vectors as their length grows?

Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, AI Engineer and Research Scientist.

Related free problems