Chapter 2: Deep learning
Batch size: gradient noise and scaling the learning rate
You will be able to say how gradient noise and cost change with the batch size, and how to scale the learning rate. Teams change the batch size to use more hardware, and a wrong learning rate then wastes the run. Interviewers ask what you change when the batch grows.
What you will learn
- Explain how gradient noise shrinks with batch size, roughly as one over its square root
- Say how to adjust the learning rate when the batch size changes
Key terms
- batch
- the group of examples whose gradients you average for one update
- epoch
- one pass over the data
- noise
- the typical error of the average
- variance
- the average squared error
- standard deviation (sd)
- the square root of the variance
Prerequisites
Take Why big nets generalize: double descent first. It comes right before this lesson in the course.
Where this lesson picks up: You saw that a bigger model can fit the training data exactly and still do well on new data, because training picks a smooth fit. Model size is one setting; the batch size is another: how does it change each step, and what should the learning rate do?
What comes next
Next, you move from training choices to model structure: convolutions, residual blocks and the arithmetic of their sizes.
Next lesson: Convolutions, residual blocks and conv arithmetic
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.