Batched Causal Self-Attention

MediumML Coding

Problem statement

Extend a self-attention mechanism to handle batched 3D inputs and apply a causal mask so each token can only attend to itself and previous tokens.

Implement the function batched_causal_self_attention(X, W_query, W_key, W_value) where:

  • X is a 3D input of shape (B, T, d_in) representing a batch of B token sequences, each of length T, with d_in-dimensional features.
  • W_query, W_key, W_value are projection matrices of shape (d_in, d_out).

Steps:

  1. Compute Q, K, V by projecting X with the three weight matrices, giving tensors of shape (B, T, d_out).
  2. Compute scaled dot-product attention scores of shape (B, T, T) using Q @ K^T divided by sqrt(d_out).
  3. Apply a causal mask: positions strictly above the main diagonal must be set to negative infinity BEFORE softmax (so they contribute zero after softmax).
  4. Apply softmax along the last axis to obtain attention weights.
  5. Compute the context vectors as attn @ V of shape (B, T, d_out).

Return the resulting context tensor as a nested Python list (use .tolist()).

Example

Input:

X = [[[1.0, 0.0], [0.0, 1.0]]] (shape (1, 2, 2))
W_query = W_key = W_value = [[1.0, 0.0], [0.0, 1.0]]

Output:

[[[1.0, 0.0], [0.3302, 0.6698]]]

Reasoning: With identity projections, Q = K = V = X. The scaled scores are X @ X^T / sqrt(2) = [[1/sqrt(2), 0], [0, 1/sqrt(2)]]. The causal mask sets the (0,1) entry to -inf, so token 0 only attends to itself (weight 1) giving context [1, 0]. Token 1 attends to both positions with scores [0, 1/sqrt(2)]; softmax gives weights approximately [0.3302, 0.6698], producing context 0.3302[1,0] + 0.6698[0,1] = [0.3302, 0.6698].

Show hint

Which three projections does attention need, and how do you compute them for every sequence in the batch at once?

Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, AI Engineer and Research Scientist.

Related free problems