Chapter 2: Deep learning
Initialization: He and Glorot from variance preservation
A deep network can fail before it learns anything, because its signal shrinks or grows a little at every layer. Interviewers ask why such a network will not train, and they expect you to derive the right size for its first weights. You will be able to compute that size for a layer and explain where it comes from.
What you will learn
- Choose He or Glorot initialization from variance preservation
Key terms
- signal
- the numbers a layer passes on to the next layer
- variance
- how spread out values are: the average squared distance from their mean
- weight
- a number a layer multiplies an input by; training changes it
- initialization
- choosing the starting values of the weights before training starts
Prerequisites
Take Normalization: BatchNorm, LayerNorm and RMSNorm first. It comes right before this lesson in the course.
Where this lesson picks up: You saw BatchNorm, LayerNorm and RMSNorm rescale activations to a steady range, layer after layer. But before training starts, how large must the weights be so that the signal does not shrink or grow on its way through?
What comes next
Initialization gets a deep network to train, but what about a run that blows up halfway, when the loss suddenly jumps or becomes NaN?
Next lesson: Debugging training: NaNs, loss spikes, clipping and mixed precision
More lessons in this part
- SGD, momentum and the learning rate 15 min
- Adam, AdamW and schedules 15 min
- Normalization: BatchNorm, LayerNorm and RMSNorm 18 min
- Debugging training: NaNs, loss spikes, clipping and mixed precision 17 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.