Long Short-Term Memory (LSTM) Network

MediumML Coding

Problem statement

Implement an LSTM that processes a sequence of inputs and returns the hidden state at every step together with the final hidden and cell states.

Write a class LSTM with:

  • __init__(self, input_size, hidden_size): random weight matrices Wf, Wi, Wc, Wo, each of shape (hidden_size, hidden_size + input_size), and zero bias column vectors bf, bi, bc, bo of shape (hidden_size, 1) (provided in the starter code).
  • forward(self, x, initial_hidden_state, initial_cell_state): x has shape (T, input_size); the states are column vectors of shape (hidden_size, 1). Return (outputs, final_h, final_c), where outputs is a numpy array stacking the hidden state of every step (shape (T, hidden_size, 1)).

At each step, form the column vector $z_t = [h_{t-1}; x_t]$ (previous hidden state on top, current input below) and compute

$$f_t = \sigma(W_f z_t + b_f), \quad i_t = \sigma(W_i z_t + b_i), \quad \tilde{c}_t = \tanh(W_c z_t + b_c), \quad o_t = \sigma(W_o z_t + b_o)$$

$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t, \qquad h_t = o_t \odot \tanh(c_t)$$

Example

Input:

import numpy as np
lstm = LSTM(input_size=1, hidden_size=1)
lstm.Wf = np.array([[0.5, 0.5]]); lstm.Wi = np.array([[0.5, 0.5]])
lstm.Wc = np.array([[0.3, 0.3]]); lstm.Wo = np.array([[0.5, 0.5]])
lstm.bf = lstm.bi = lstm.bc = lstm.bo = np.array([[0.1]])
outputs, final_h, final_c = lstm.forward(np.array([[1.0], [2.0], [3.0]]), np.zeros((1, 1)), np.zeros((1, 1)))
print(final_h)

Output:

[[0.73698596]]

Reasoning: the cell runs three steps; each step combines the previous hidden state and the current input through the four gates, and the final hidden state is $o_3 \tanh(c_3)$.

Show hint

Which gates does an LSTM compute at every step, and which activation function belongs to each?

Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.

Related free problems