Chapter 2: Deep learning

Why big nets generalize: double descent

You will be able to sketch double descent and locate the point where exact training fit first becomes possible. This helps you explain why a huge network can predict well on new examples.

18 minInteractive lessonChapter 2: Deep learningLesson 8 of 20

What you will learn

  • Describe the double-descent curve and where the interpolation threshold sits

Key terms

model size
the number of adjustable parameters in a model
test error
how wrong the model is on new points it did not train on
underfit
miss the pattern because the model is too simple
overfit
follow the noise in the training points and predict new points badly

Prerequisites

Take Debugging training: NaNs, loss spikes, clipping and mixed precision first. It comes right before this lesson in the course.

Where this lesson picks up: You found causes of unstable training and ways to control them. A run that trains well leaves another question: if a model fits its training points exactly, must it predict badly?

What comes next

How does batch size change gradient descent, and what should happen to the learning rate?

Next lesson: Batch size: gradient noise and scaling the learning rate

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.