Chapter 2: Deep learning
Adam, AdamW and schedules
Use gradient history to scale each weight, repair averages that start at zero, and combine weight decay with a changing learning rate.
What you will learn
- Write one AdamW update including bias correction
- Explain why decoupled weight decay differs from an L2 penalty under Adam
- Implement a warmup-plus-cosine learning-rate schedule
Key terms
- Adam
- an optimizer that scales each weight using its own gradient history
- AdamW
- Adam with a separate weight shrink
- schedule
- a rule that changes the learning rate over training steps
Prerequisites
Take SGD, momentum and the learning rate first. It comes right before this lesson in the course.
Where this lesson picks up: SGD scales gradients by a learning rate; momentum smooths them with a running average.
What comes next
Next, normalization controls the scale of values inside a layer.
Next lesson: Normalization: BatchNorm, LayerNorm and RMSNorm
More lessons in this part
- SGD, momentum and the learning rate 15 min
- Normalization: BatchNorm, LayerNorm and RMSNorm 18 min
- Initialization: He and Glorot from variance preservation 18 min
- Debugging training: NaNs, loss spikes, clipping and mixed precision 17 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.