Neural networks

Recurrent network

Imagine reading a story while keeping a short note about what has happened. An RNN updates a hidden state whenever it reads the next item. The same update rule is reused at every step, letting earlier inputs influence later outputs.

Step 1x = 1; h = 1Step 2x = 2; h = 2.5Step 3x = 0; h = 1.25
Toy linear recurrence h = 0.5 × previous h + x, starting at h = 0; practical RNNs usually add an activation.
Step 1x = 1; h = 1Step 2x = 2; h = 2.5Step 3x = 0; h = 1.25
Toy linear recurrence h = 0.5 × previous h + x, starting at h = 0; practical RNNs usually add an activation.

How it works

  1. Initialize the hidden state, often to zeros.
  2. At each time step, combine the current input and previous hidden state, then apply a nonlinear activation.
  3. Use selected hidden states to predict outputs for the sequence or individual steps.
  4. Unroll the computation through time and backpropagate, optionally truncating how far gradients travel.

$$h_t=\tanh(W_xx_t+W_hh_{t-1}+b)$$

The knobs that matter

hidden size
sets the size of the state carried between steps
number of layers
adds recurrent transformations and computation
truncation length
limits how far training gradients travel through time
gradient clipping threshold
caps large gradients to help control exploding updates

What interviewers ask

  • How does an RNN differ from an MLP or a CNN?
  • Why do gradients vanish or explode in a simple RNN?
  • How does backpropagation through time work?
  • When would you choose an LSTM instead of a simple RNN?

Common mistakes

  • Failing to reset state between unrelated sequences leaks information across examples.
  • Clipping can help exploding gradients but does not fix vanishing gradients.
  • Padding tokens can affect states and losses unless sequence lengths or masks are handled correctly.

When to use it

Use an RNN for sequential updates when a compact running state fits the task. Simple RNNs struggle with long dependencies, and their recurrent steps limit parallelism across time.

Implement it (2 problems)

Learn it

Interview questions about recurrent network