Scaled Dot-Product Attention

MediumLLMs & GenAI

Reported in interviews at Meta

Problem statement

Implement scaled dot-product attention, the core building block of Transformer architectures. Given precomputed query, key and value matrices, compute the attention output and the attention weight matrix.

Your function should:

  1. Compute attention scores from Q and K with the appropriate scaling.
  2. Optionally apply a mask before computing the weights (mask value 0 blocks a position, 1 keeps it).
  3. Apply a numerically stable softmax to get the attention weights.
  4. Compute the weighted sum of the values.

Write scaled_dot_product_attention(Q, K, V, mask=None) returning a tuple (output, attention_weights).

Inputs:

  • Q: shape (seq_len_q, d_k).
  • K: shape (seq_len_k, d_k).
  • V: shape (seq_len_k, d_v).
  • mask: optional binary array of shape (seq_len_q, seq_len_k); 0 = block, 1 = attend.

Outputs:

  • output with shape (seq_len_q, d_v).
  • attention_weights with shape (seq_len_q, seq_len_k).

Example

Input:

Q = [[1.0, 0.0], [0.0, 1.0]]
K = [[1.0, 0.0], [0.0, 1.0]]
V = [[1.0, 2.0], [3.0, 4.0]]

Output:

output = [[1.6605, 2.6605], [2.3395, 3.3395]]
weights = [[0.6698, 0.3302], [0.3302, 0.6698]]

Reasoning: with $d_k = 2$ the scores are $QK^T/\sqrt{2} = [[0.7071, 0], [0, 0.7071]]$. The softmax of row 0 is $[0.6698, 0.3302]$, so the first output row is $0.6698 \cdot [1, 2] + 0.3302 \cdot [3, 4] = [1.6605, 2.6605]$.

Show hint

How do you turn similarity scores between each query and every key into a distribution over keys?

Relevant roles: ML Engineer, LLM Engineer, AI Engineer, Deep Learning Engineer and Research Scientist.

Related free problems