Chapter 2: Deep learning
Debugging training: NaNs, loss spikes, clipping and mixed precision
A model can stop learning halfway through a run. You need evidence before you choose a repair, both at work and in interviews.
What you will learn
- Triage NaNs and loss spikes with gradient norms and clipping
- Explain FP16 vs BF16 range and why loss scaling exists
Key terms
- gradient
- a suggested change along a direction
- vector
- an ordered list of numbers
- global gradient norm
- a measure of the combined length of all gradients
Prerequisites
Take Initialization: He and Glorot from variance preservation first. It comes right before this lesson in the course.
Where this lesson picks up: You will be able to diagnose a failed run and explain why small gradients need loss scaling. You previously kept signals stable across layers; what happens when a run later fails?
What comes next
Once training works, why can a larger model do better on new examples?
Next lesson: Why big nets generalize: double descent
More lessons in this part
- SGD, momentum and the learning rate 15 min
- Adam, AdamW and schedules 15 min
- Normalization: BatchNorm, LayerNorm and RMSNorm 18 min
- Initialization: He and Glorot from variance preservation 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.