Chapter 2: Deep learning
Why big nets generalize: double descent
You will be able to sketch double descent and locate the point where exact training fit first becomes possible. This helps you explain why a huge network can predict well on new examples.
What you will learn
- Describe the double-descent curve and where the interpolation threshold sits
Key terms
- model size
- the number of adjustable parameters in a model
- test error
- how wrong the model is on new points it did not train on
- underfit
- miss the pattern because the model is too simple
- overfit
- follow the noise in the training points and predict new points badly
Prerequisites
Take Debugging training: NaNs, loss spikes, clipping and mixed precision first. It comes right before this lesson in the course.
Where this lesson picks up: You found causes of unstable training and ways to control them. A run that trains well leaves another question: if a model fits its training points exactly, must it predict badly?
What comes next
How does batch size change gradient descent, and what should happen to the learning rate?
Next lesson: Batch size: gradient noise and scaling the learning rate
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.