Chapter 2: Deep learning

Debugging training: NaNs, loss spikes, clipping and mixed precision

A model can stop learning halfway through a run. You need evidence before you choose a repair, both at work and in interviews.

17 minInteractive lessonChapter 2: Deep learningLesson 7 of 20

What you will learn

  • Triage NaNs and loss spikes with gradient norms and clipping
  • Explain FP16 vs BF16 range and why loss scaling exists

Key terms

gradient
a suggested change along a direction
vector
an ordered list of numbers
global gradient norm
a measure of the combined length of all gradients

Prerequisites

Take Initialization: He and Glorot from variance preservation first. It comes right before this lesson in the course.

Where this lesson picks up: You will be able to diagnose a failed run and explain why small gradients need loss scaling. You previously kept signals stable across layers; what happens when a run later fails?

What comes next

Once training works, why can a larger model do better on new examples?

Next lesson: Why big nets generalize: double descent

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.