Chapter 2: Deep learning

RNNs, LSTMs and why gradients vanish

You will be able to explain why a training signal fades or explodes over many steps. You will also be able to write the forward pass of a long short-term memory (LSTM) cell. Interviewers for machine-learning engineer roles often ask why plain recurrent networks fail on long-range links and how an LSTM helps.

7 minInteractive lessonChapter 2: Deep learningLesson 13 of 20

What you will learn

  • Write an LSTM cell forward pass
  • Explain vanishing gradients through repeated Jacobians
  • Explain how the forget gate sets a memory horizon

Key terms

error signal
the number sent back through the steps to update the weights
recurrent network (RNN)
a network that reads one step at a time and reuses the same unit
gate
a number from 0 to 1 that multiplies a flow
LSTM
a recurrent unit with gates and a memory that adds new values

Prerequisites

Take CNNs vs ViTs and transfer learning first. It comes right before this lesson in the course.

Where this lesson picks up: You saw that a vision transformer (ViT) looks at all its image patches at once and needs more data or pretraining than a CNN. Text often arrives one word at a time, so how can a model carry a memory from one word to the next?

What comes next

The LSTM reads one word vector per step. Where do good word vectors come from?

Next lesson: Contrastive learning: word2vec, InfoNCE and CLIP

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.