Chapter 2: Deep learning
RNNs, LSTMs and why gradients vanish
You will be able to explain why a training signal fades or explodes over many steps. You will also be able to write the forward pass of a long short-term memory (LSTM) cell. Interviewers for machine-learning engineer roles often ask why plain recurrent networks fail on long-range links and how an LSTM helps.
What you will learn
- Write an LSTM cell forward pass
- Explain vanishing gradients through repeated Jacobians
- Explain how the forget gate sets a memory horizon
Key terms
- error signal
- the number sent back through the steps to update the weights
- recurrent network (RNN)
- a network that reads one step at a time and reuses the same unit
- gate
- a number from 0 to 1 that multiplies a flow
- LSTM
- a recurrent unit with gates and a memory that adds new values
Prerequisites
Take CNNs vs ViTs and transfer learning first. It comes right before this lesson in the course.
Where this lesson picks up: You saw that a vision transformer (ViT) looks at all its image patches at once and needs more data or pretraining than a CNN. Text often arrives one word at a time, so how can a model carry a memory from one word to the next?
What comes next
The LSTM reads one word vector per step. Where do good word vectors come from?
Next lesson: Contrastive learning: word2vec, InfoNCE and CLIP
More lessons in this part
- Contrastive learning: word2vec, InfoNCE and CLIP 7 min
- Why transformers replaced RNNs: parallelism and teacher forcing 22 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.