Sliding Window Attention
Problem statement
Implement sliding window (local) attention, where each query position attends only to key positions in a fixed neighborhood instead of the whole sequence.
Write sliding_window_attention(Q, K, V, window_size) where Q and K have shape (seq_len, d_k), V has shape (seq_len, d_v), and window_size is $w$. Position $i$ may attend to position $j$ only if $|i - j| \le w$.
- Compute the scores $S = QK^\top / \sqrt{d_k}$.
- Set every score outside the window to $-\infty$.
- Apply a numerically stable row-wise softmax (subtract the row max).
- Multiply the attention weights by
V.
Return the output of shape (seq_len, d_v) rounded to 4 decimal places. With window_size = 1, position 2 of a length-5 sequence sees positions 1, 2 and 3; with window_size = 0, each position sees only itself.
Example
Input:
Q = np.array([[1.0], [2.0], [3.0]])
K = np.array([[1.0], [2.0], [3.0]])
V = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
sliding_window_attention(Q, K, V, window_size=1)
Output:
[[0.2689, 0.7311], [0.8827, 0.9841], [0.9526, 1.0]]
Reasoning: with $d_k = 1$ the scores are [[1, 2, 3], [2, 4, 6], [3, 6, 9]]. The window masks row 0 to [1, 2, -inf] and row 2 to [-inf, 6, 9]; row 1 keeps all three. Row-wise softmax gives [0.2689, 0.7311, 0], [0.0159, 0.1173, 0.8668] and [0, 0.0474, 0.9526], and multiplying by V gives the output above.
Show hint
How can you stop a softmax from giving any weight to certain positions while keeping the remaining weights normalized?
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, AI Engineer and Research Scientist.
Related free problems
- Gated AttentionMedium
- Masked Self-AttentionMedium
- Multi-Head AttentionMedium