Bahdanau Additive Attention
Problem statement
The original attention mechanism for encoder-decoder RNNs (Bahdanau et al., 2015) scores each encoder state against the decoder's current state with a tiny neural network rather than a dot product:
$$e_i = v^\top \tanh(W_q\, q + W_k\, k_i), \qquad \alpha_i = \frac{\exp(e_i)}{\sum_j \exp(e_j)}, \qquad c = \sum_i \alpha_i\, k_i$$
Implement additive_attention(query, keys, W_q, W_k, v):
query:(d_q,)decoder state;keys:(n, d_k)encoder states (they also serve as the values)W_q:(d_a, d_q),W_k:(d_a, d_k),v:(d_a,)
Return (weights, context) where weights has shape (n,) and sums to 1, and context has shape (d_k,). Use a numerically stable softmax (subtract the maximum score before exponentiating).
Example
Input:
w, c = additive_attention(np.array([0.5]), np.array([[0.5], [-0.5]]),
np.array([[1.0], [1.0]]), np.array([[1.0], [-1.0]]),
np.array([2.0, 0.0]))
Output:
weights = [0.821, 0.179] (rounded)
context = [0.321]
Reasoning: in the first hidden unit the pre-activations are $0.5 + 0.5 = 1$ for key 1 and $0.5 - 0.5 = 0$ for key 2, and $v$ selects only that unit, so the scores are $[2\tanh 1, 0] \approx [1.523, 0]$. Softmax gives about $[0.821, 0.179]$, and the context is $0.821 \cdot 0.5 + 0.179 \cdot (-0.5) \approx 0.321$.
Show hint
How can a tiny network compare two vectors of different sizes and output a single relevance number?
Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer, Research Scientist and Applied Scientist.
Related free problems
- Gated AttentionMedium
- Multi-Head AttentionMedium