Masked Self-Attention

MediumML Coding

Reported in interviews at Scale AI

Problem statement

Implement masked_attention(Q, K, V, mask) using NumPy. Each token may attend only to itself and earlier tokens, as specified by the supplied mask.

Use scaled dot-product attention:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V, \qquad d_k = Q.\text{shape}[1]$$

where the softmax is taken over each row.

Inputs:

  • Q, K, V: np.ndarray of shape (seq_len, d_model)
  • mask: np.ndarray of shape (seq_len, seq_len), with 0 at allowed positions and -inf at blocked positions (every row has at least one allowed position)

Output:

  • np.ndarray of shape (seq_len, d_model)

Example

Input:

Q = K = V = np.array([[1.0, 0.0], [0.0, 1.0]])
mask = np.triu(np.ones((2, 2)) * (-np.inf), k=1)
print(np.round(masked_attention(Q, K, V, mask), 4))

Output:

[[1.     0.    ]
 [0.3302 0.6698]]

Reasoning: Row 0 may only attend to itself, so it copies $V_0$. Row 1 sees scores $[0, 1/\sqrt{2}]$, whose softmax is $[0.3302, 0.6698]$, giving that mix of $V_0$ and $V_1$.

Show hint

What quantity measures how well each query matches each key, and how should it be scaled?

Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.

Related free problems