Chapter 2: Deep learning

Why transformers replaced RNNs: parallelism and teacher forcing

"Why did transformers replace RNNs?" is a standard question in ML engineer interviews. A strong answer gives numbers. It says how many steps an RNN waits and how far apart two tokens are. It also says what training stores in memory, and why a model that trains well can still drift in use.

22 minInteractive lessonChapter 2: Deep learningLesson 15 of 20

What you will learn

  • Compare RNN, CNN and self-attention on path length and parallelism
  • Explain teacher forcing and exposure bias
  • Explain the cost of truncated backpropagation through time

Key terms

token
one word or a piece of a word
hidden state
the numbers an RNN passes from one token to the next
self-attention
the layer that scores every pair of tokens at once
score
one number: how much one token looks at another
transformer
a model built from attention layers

Prerequisites

Take Contrastive learning: word2vec, InfoNCE and CLIP first. It comes right before this lesson in the course.

Where this lesson picks up: Last lesson, you scored every pair in a batch at once, in one softmax. Can a model read a whole sentence that way, all pairs at once, instead of one token at a time?

What comes next

A sentence is a chain of tokens. What if the links can form any shape?

Next lesson: Graphs as matrices: PageRank and spectral clustering

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.