Batched Causal Self-Attention
Problem statement
Extend a self-attention mechanism to handle batched 3D inputs and apply a causal mask so each token can only attend to itself and previous tokens.
Implement the function batched_causal_self_attention(X, W_query, W_key, W_value) where:
Xis a 3D input of shape(B, T, d_in)representing a batch ofBtoken sequences, each of lengthT, withd_in-dimensional features.W_query,W_key,W_valueare projection matrices of shape(d_in, d_out).
Steps:
- Compute Q, K, V by projecting
Xwith the three weight matrices, giving tensors of shape(B, T, d_out). - Compute scaled dot-product attention scores of shape
(B, T, T)usingQ @ K^Tdivided bysqrt(d_out). - Apply a causal mask: positions strictly above the main diagonal must be set to negative infinity BEFORE softmax (so they contribute zero after softmax).
- Apply softmax along the last axis to obtain attention weights.
- Compute the context vectors as
attn @ Vof shape(B, T, d_out).
Return the resulting context tensor as a nested Python list (use .tolist()).
Example
Input:
X = [[[1.0, 0.0], [0.0, 1.0]]] (shape (1, 2, 2))
W_query = W_key = W_value = [[1.0, 0.0], [0.0, 1.0]]
Output:
[[[1.0, 0.0], [0.3302, 0.6698]]]
Reasoning: With identity projections, Q = K = V = X. The scaled scores are X @ X^T / sqrt(2) = [[1/sqrt(2), 0], [0, 1/sqrt(2)]]. The causal mask sets the (0,1) entry to -inf, so token 0 only attends to itself (weight 1) giving context [1, 0]. Token 1 attends to both positions with scores [0, 1/sqrt(2)]; softmax gives weights approximately [0.3302, 0.6698], producing context 0.3302[1,0] + 0.6698[0,1] = [0.3302, 0.6698].
Show hint
Which three projections does attention need, and how do you compute them for every sequence in the batch at once?
Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, AI Engineer and Research Scientist.
Related free problems
- Masked Self-AttentionMedium
- Gated AttentionMedium
- Multi-Head AttentionMedium