Chapter 2: Deep learning

SGD, momentum and the learning rate

Choose a step size, smooth noisy gradients and spot steps that bounce past a minimum.

15 minInteractive lessonChapter 2: Deep learningLesson 4

What you will learn

  • Compute a plain gradient step using a learning rate
  • Update momentum as a running average of batch gradients
  • Explain how the learning rate changes stability and progress

Key terms

gradient
the local slope of loss as a weight increases
learning rate
the multiplier that scales a gradient into a step
momentum
a running average of recent gradients

Prerequisites

Take Backprop by hand on a two-layer MLP first. It comes right before this lesson in the course.

Where this lesson picks up: Backprop gives a gradient for every weight. Now turn those gradients into useful steps.

What comes next

Next, Adam gives each weight its own scale using its gradient history.

Next lesson: Adam, AdamW and schedules

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.