Neural networks
LSTM and GRU
Imagine a notebook with controls for erasing old notes, writing new ones and reading what matters. An LSTM learns gates that control these operations on a memory cell. A GRU uses a related gated update with one state instead of separate cell and hidden states.
How it works
- Combine the current input and previous hidden state to calculate gates and candidate content.
- In an LSTM, multiply the old cell by the forget gate and add the input-gated candidate.
- Use the output gate and transformed cell to produce the new hidden state; a GRU instead blends its previous state with a candidate using an update gate.
- Train through time, using masks and gradient clipping where needed.
$$c_t=f_t\odot c_{t-1}+i_t\odot\tilde{c}_t,\qquad h_t=o_t\odot\tanh(c_t)$$
The knobs that matter
- hidden size
- larger states store more features and cost more per step
- cell type
- GRU uses fewer gate computations than a comparable standard LSTM
- bidirectionality
- adds future context but prevents strictly causal streaming
- sequence length
- sets the context and gradient path seen during training
What interviewers ask
- Can you explain the gates in an LSTM?
- How does a GRU differ from an LSTM?
- Why do gates help with vanishing gradients?
- What would you consider when serving an LSTM with low latency?
Common mistakes
- Gates help gradient flow but do not guarantee unlimited memory or prevent all vanishing gradients.
- A bidirectional model uses future inputs and cannot be treated as a causal streaming model.
- Confusing the cell state with the hidden state leads to incorrect LSTM implementations.
When to use it
Use gated recurrent models for sequence tasks and streaming systems where a fixed-size state is useful. Their sequential computation can be limiting when training on long sequences.
Implement it (3 problems)
- GRU Cell Forward PassProEasySolve
Learn it
Interview questions about lstm and gru
- LSTM vs GRUProEasySolve
- How an LSTM WorksProMediumSolve