Masked Self-Attention
Problem statement
Implement masked_attention(Q, K, V, mask) using NumPy. Each token may attend only to itself and earlier tokens, as specified by the supplied mask.
Use scaled dot-product attention:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V, \qquad d_k = Q.\text{shape}[1]$$
where the softmax is taken over each row.
Inputs:
Q,K,V: np.ndarray of shape(seq_len, d_model)mask: np.ndarray of shape(seq_len, seq_len), with0at allowed positions and-infat blocked positions (every row has at least one allowed position)
Output:
- np.ndarray of shape
(seq_len, d_model)
Example
Input:
Q = K = V = np.array([[1.0, 0.0], [0.0, 1.0]])
mask = np.triu(np.ones((2, 2)) * (-np.inf), k=1)
print(np.round(masked_attention(Q, K, V, mask), 4))
Output:
[[1. 0. ]
[0.3302 0.6698]]
Reasoning: Row 0 may only attend to itself, so it copies $V_0$. Row 1 sees scores $[0, 1/\sqrt{2}]$, whose softmax is $[0.3302, 0.6698]$, giving that mix of $V_0$ and $V_1$.
Show hint
What quantity measures how well each query matches each key, and how should it be scaled?
Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.
Related free problems
- Sliding Window AttentionMedium
- Gated AttentionMedium