Chapter 2: Deep learning
SGD, momentum and the learning rate
Choose a step size, smooth noisy gradients and spot steps that bounce past a minimum.
What you will learn
- Compute a plain gradient step using a learning rate
- Update momentum as a running average of batch gradients
- Explain how the learning rate changes stability and progress
Key terms
- gradient
- the local slope of loss as a weight increases
- learning rate
- the multiplier that scales a gradient into a step
- momentum
- a running average of recent gradients
Prerequisites
Take Backprop by hand on a two-layer MLP first. It comes right before this lesson in the course.
Where this lesson picks up: Backprop gives a gradient for every weight. Now turn those gradients into useful steps.
What comes next
Next, Adam gives each weight its own scale using its gradient history.
Next lesson: Adam, AdamW and schedules
More lessons in this part
- Adam, AdamW and schedules 15 min
- Normalization: BatchNorm, LayerNorm and RMSNorm 18 min
- Initialization: He and Glorot from variance preservation 18 min
- Debugging training: NaNs, loss spikes, clipping and mixed precision 17 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.