Chapter 2: Deep learning

Initialization: He and Glorot from variance preservation

A deep network can fail before it learns anything, because its signal shrinks or grows a little at every layer. Interviewers ask why such a network will not train, and they expect you to derive the right size for its first weights. You will be able to compute that size for a layer and explain where it comes from.

18 minInteractive lessonChapter 2: Deep learningLesson 6 of 20

What you will learn

  • Choose He or Glorot initialization from variance preservation

Key terms

signal
the numbers a layer passes on to the next layer
variance
how spread out values are: the average squared distance from their mean
weight
a number a layer multiplies an input by; training changes it
initialization
choosing the starting values of the weights before training starts

Prerequisites

Take Normalization: BatchNorm, LayerNorm and RMSNorm first. It comes right before this lesson in the course.

Where this lesson picks up: You saw BatchNorm, LayerNorm and RMSNorm rescale activations to a steady range, layer after layer. But before training starts, how large must the weights be so that the signal does not shrink or grow on its way through?

What comes next

Initialization gets a deep network to train, but what about a run that blows up halfway, when the loss suddenly jumps or becomes NaN?

Next lesson: Debugging training: NaNs, loss spikes, clipping and mixed precision

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.