Neural networks
Recurrent network
Imagine reading a story while keeping a short note about what has happened. An RNN updates a hidden state whenever it reads the next item. The same update rule is reused at every step, letting earlier inputs influence later outputs.
How it works
- Initialize the hidden state, often to zeros.
- At each time step, combine the current input and previous hidden state, then apply a nonlinear activation.
- Use selected hidden states to predict outputs for the sequence or individual steps.
- Unroll the computation through time and backpropagate, optionally truncating how far gradients travel.
$$h_t=\tanh(W_xx_t+W_hh_{t-1}+b)$$
The knobs that matter
- hidden size
- sets the size of the state carried between steps
- number of layers
- adds recurrent transformations and computation
- truncation length
- limits how far training gradients travel through time
- gradient clipping threshold
- caps large gradients to help control exploding updates
What interviewers ask
- How does an RNN differ from an MLP or a CNN?
- Why do gradients vanish or explode in a simple RNN?
- How does backpropagation through time work?
- When would you choose an LSTM instead of a simple RNN?
Common mistakes
- Failing to reset state between unrelated sequences leaks information across examples.
- Clipping can help exploding gradients but does not fix vanishing gradients.
- Padding tokens can affect states and losses unless sequence lengths or masks are handled correctly.
When to use it
Use an RNN for sequential updates when a compact running state fits the task. Simple RNNs struggle with long dependencies, and their recurrent steps limit parallelism across time.
Implement it (2 problems)
- Simple RNN Forward PassProEasySolve
Learn it
Interview questions about recurrent network
- MLP vs CNN vs RNNProEasySolve