Neural networks

LSTM and GRU

Imagine a notebook with controls for erasing old notes, writing new ones and reading what matters. An LSTM learns gates that control these operations on a memory cell. A GRU uses a related gated update with one state instead of separate cell and hidden states.

Old cellc = 2Keep and write0.9 × 2 + 0.2 × 0.5New cellc = 1.9
The forget gate keeps 0.9 × 2 = 1.8; the input gate writes 0.2 × 0.5 = 0.1, giving a new cell of 1.9.
Old cellc = 2Keep and write0.9 × 2 + 0.2 × 0.5New cellc = 1.9
The forget gate keeps 0.9 × 2 = 1.8; the input gate writes 0.2 × 0.5 = 0.1, giving a new cell of 1.9.

How it works

  1. Combine the current input and previous hidden state to calculate gates and candidate content.
  2. In an LSTM, multiply the old cell by the forget gate and add the input-gated candidate.
  3. Use the output gate and transformed cell to produce the new hidden state; a GRU instead blends its previous state with a candidate using an update gate.
  4. Train through time, using masks and gradient clipping where needed.

$$c_t=f_t\odot c_{t-1}+i_t\odot\tilde{c}_t,\qquad h_t=o_t\odot\tanh(c_t)$$

The knobs that matter

hidden size
larger states store more features and cost more per step
cell type
GRU uses fewer gate computations than a comparable standard LSTM
bidirectionality
adds future context but prevents strictly causal streaming
sequence length
sets the context and gradient path seen during training

What interviewers ask

  • Can you explain the gates in an LSTM?
  • How does a GRU differ from an LSTM?
  • Why do gates help with vanishing gradients?
  • What would you consider when serving an LSTM with low latency?

Common mistakes

  • Gates help gradient flow but do not guarantee unlimited memory or prevent all vanishing gradients.
  • A bidirectional model uses future inputs and cannot be treated as a causal streaming model.
  • Confusing the cell state with the hidden state leads to incorrect LSTM implementations.

When to use it

Use gated recurrent models for sequence tasks and streaming systems where a fixed-size state is useful. Their sequential computation can be limiting when training on long sequences.

Implement it (3 problems)

Learn it

Interview questions about lstm and gru