Gated Attention

MediumML Coding

Problem statement

Implement gated attention, a small change to scaled dot-product attention that adds a query-dependent output gate.

In standard attention the softmax forces the weights for each query to sum to 1, so a head must put its attention somewhere even when no token is relevant; this is one cause of "attention sinks" on the first token. Gated attention multiplies the attention output element-wise by a sigmoid gate computed from the input, letting each position suppress an uninformative output:

$$Y = \text{softmax}\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V, \qquad G = \sigma(X W_g), \qquad Y_{\text{gated}} = G \odot Y,$$

with $Q = X W_q$, $K = X W_k$, $V = X W_v$.

Write gated_attention(X, W_q, W_k, W_v, W_g) where X has shape (seq_len, d_model), W_q and W_k have shape (d_model, d_k), and W_v and W_g have shape (d_model, d_v). Return the (seq_len, d_v) output rounded to 4 decimal places. Use a numerically stable softmax; inputs may produce large scores.

Example

Input:

X = [[1, 0], [0, 1]], W_q = W_k = W_v = I (identity), W_g = zeros

Output:

[[0.3349, 0.1651], [0.1651, 0.3349]]

Reasoning: With $Q = K = V = X = I$ and $d_k = 2$, the scores are $I / \sqrt{2}$, so each row of the softmax puts weight $e^{0.7071} / (e^{0.7071} + 1) \approx 0.6698$ on itself and $0.3302$ on the other token. With $W_g = 0$ the gate is $\sigma(0) = 0.5$ everywhere, halving the attention output to $[[0.3349, 0.1651], [0.1651, 0.3349]]$.

Show hint

What does standard scaled dot-product attention compute before any gating is involved?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, Research Scientist and Gen AI Engineer.

Related free problems