Bahdanau Additive Attention

MediumML Coding

Problem statement

The original attention mechanism for encoder-decoder RNNs (Bahdanau et al., 2015) scores each encoder state against the decoder's current state with a tiny neural network rather than a dot product:

$$e_i = v^\top \tanh(W_q\, q + W_k\, k_i), \qquad \alpha_i = \frac{\exp(e_i)}{\sum_j \exp(e_j)}, \qquad c = \sum_i \alpha_i\, k_i$$

Implement additive_attention(query, keys, W_q, W_k, v):

  • query: (d_q,) decoder state; keys: (n, d_k) encoder states (they also serve as the values)
  • W_q: (d_a, d_q), W_k: (d_a, d_k), v: (d_a,)

Return (weights, context) where weights has shape (n,) and sums to 1, and context has shape (d_k,). Use a numerically stable softmax (subtract the maximum score before exponentiating).

Example

Input:

w, c = additive_attention(np.array([0.5]), np.array([[0.5], [-0.5]]),
                          np.array([[1.0], [1.0]]), np.array([[1.0], [-1.0]]),
                          np.array([2.0, 0.0]))

Output:

weights = [0.821, 0.179]   (rounded)
context = [0.321]

Reasoning: in the first hidden unit the pre-activations are $0.5 + 0.5 = 1$ for key 1 and $0.5 - 0.5 = 0$ for key 2, and $v$ selects only that unit, so the scores are $[2\tanh 1, 0] \approx [1.523, 0]$. Softmax gives about $[0.821, 0.179]$, and the context is $0.821 \cdot 0.5 + 0.179 \cdot (-0.5) \approx 0.321$.

Show hint

How can a tiny network compare two vectors of different sizes and output a single relevance number?

Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer, Research Scientist and Applied Scientist.

Related free problems