Sliding Window Attention

MediumML Coding

Problem statement

Implement sliding window (local) attention, where each query position attends only to key positions in a fixed neighborhood instead of the whole sequence.

Write sliding_window_attention(Q, K, V, window_size) where Q and K have shape (seq_len, d_k), V has shape (seq_len, d_v), and window_size is $w$. Position $i$ may attend to position $j$ only if $|i - j| \le w$.

  1. Compute the scores $S = QK^\top / \sqrt{d_k}$.
  2. Set every score outside the window to $-\infty$.
  3. Apply a numerically stable row-wise softmax (subtract the row max).
  4. Multiply the attention weights by V.

Return the output of shape (seq_len, d_v) rounded to 4 decimal places. With window_size = 1, position 2 of a length-5 sequence sees positions 1, 2 and 3; with window_size = 0, each position sees only itself.

Example

Input:

Q = np.array([[1.0], [2.0], [3.0]])
K = np.array([[1.0], [2.0], [3.0]])
V = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
sliding_window_attention(Q, K, V, window_size=1)

Output:

[[0.2689, 0.7311], [0.8827, 0.9841], [0.9526, 1.0]]

Reasoning: with $d_k = 1$ the scores are [[1, 2, 3], [2, 4, 6], [3, 6, 9]]. The window masks row 0 to [1, 2, -inf] and row 2 to [-inf, 6, 9]; row 1 keeps all three. Row-wise softmax gives [0.2689, 0.7311, 0], [0.0159, 0.1173, 0.8668] and [0, 0.0474, 0.9526], and multiplying by V gives the output above.

Show hint

How can you stop a softmax from giving any weight to certain positions while keeping the remaining weights normalized?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, AI Engineer and Research Scientist.

Related free problems