Chapter 2: Deep learning

Batch size: gradient noise and scaling the learning rate

You will be able to say how gradient noise and cost change with the batch size, and how to scale the learning rate. Teams change the batch size to use more hardware, and a wrong learning rate then wastes the run. Interviewers ask what you change when the batch grows.

18 minInteractive lessonChapter 2: Deep learningLesson 9 of 20

What you will learn

  • Explain how gradient noise shrinks with batch size, roughly as one over its square root
  • Say how to adjust the learning rate when the batch size changes

Key terms

batch
the group of examples whose gradients you average for one update
epoch
one pass over the data
noise
the typical error of the average
variance
the average squared error
standard deviation (sd)
the square root of the variance

Prerequisites

Take Why big nets generalize: double descent first. It comes right before this lesson in the course.

Where this lesson picks up: You saw that a bigger model can fit the training data exactly and still do well on new data, because training picks a smooth fit. Model size is one setting; the batch size is another: how does it change each step, and what should the learning rate do?

What comes next

Next, you move from training choices to model structure: convolutions, residual blocks and the arithmetic of their sizes.

Next lesson: Convolutions, residual blocks and conv arithmetic

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.