Long Short-Term Memory (LSTM) Network
Problem statement
Implement an LSTM that processes a sequence of inputs and returns the hidden state at every step together with the final hidden and cell states.
Write a class LSTM with:
__init__(self, input_size, hidden_size): random weight matricesWf,Wi,Wc,Wo, each of shape(hidden_size, hidden_size + input_size), and zero bias column vectorsbf,bi,bc,boof shape(hidden_size, 1)(provided in the starter code).forward(self, x, initial_hidden_state, initial_cell_state):xhas shape(T, input_size); the states are column vectors of shape(hidden_size, 1). Return(outputs, final_h, final_c), whereoutputsis a numpy array stacking the hidden state of every step (shape(T, hidden_size, 1)).
At each step, form the column vector $z_t = [h_{t-1}; x_t]$ (previous hidden state on top, current input below) and compute
$$f_t = \sigma(W_f z_t + b_f), \quad i_t = \sigma(W_i z_t + b_i), \quad \tilde{c}_t = \tanh(W_c z_t + b_c), \quad o_t = \sigma(W_o z_t + b_o)$$
$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t, \qquad h_t = o_t \odot \tanh(c_t)$$
Example
Input:
import numpy as np
lstm = LSTM(input_size=1, hidden_size=1)
lstm.Wf = np.array([[0.5, 0.5]]); lstm.Wi = np.array([[0.5, 0.5]])
lstm.Wc = np.array([[0.3, 0.3]]); lstm.Wo = np.array([[0.5, 0.5]])
lstm.bf = lstm.bi = lstm.bc = lstm.bo = np.array([[0.1]])
outputs, final_h, final_c = lstm.forward(np.array([[1.0], [2.0], [3.0]]), np.zeros((1, 1)), np.zeros((1, 1)))
print(final_h)
Output:
[[0.73698596]]
Reasoning: the cell runs three steps; each step combines the previous hidden state and the current input through the four gates, and the final hidden state is $o_3 \tanh(c_3)$.
Show hint
Which gates does an LSTM compute at every step, and which activation function belongs to each?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium