Chapter 2: Deep learning
Why transformers replaced RNNs: parallelism and teacher forcing
"Why did transformers replace RNNs?" is a standard question in ML engineer interviews. A strong answer gives numbers. It says how many steps an RNN waits and how far apart two tokens are. It also says what training stores in memory, and why a model that trains well can still drift in use.
What you will learn
- Compare RNN, CNN and self-attention on path length and parallelism
- Explain teacher forcing and exposure bias
- Explain the cost of truncated backpropagation through time
Key terms
- token
- one word or a piece of a word
- hidden state
- the numbers an RNN passes from one token to the next
- self-attention
- the layer that scores every pair of tokens at once
- score
- one number: how much one token looks at another
- transformer
- a model built from attention layers
Prerequisites
Take Contrastive learning: word2vec, InfoNCE and CLIP first. It comes right before this lesson in the course.
Where this lesson picks up: Last lesson, you scored every pair in a batch at once, in one softmax. Can a model read a whole sentence that way, all pairs at once, instead of one token at a time?
What comes next
A sentence is a chain of tokens. What if the links can form any shape?
Next lesson: Graphs as matrices: PageRank and spectral clustering
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.