Chapter 2: Deep learning
Normalization: BatchNorm, LayerNorm and RMSNorm
A normalization layer divides a row of numbers by its size, so every layer gets inputs on the same scale. Interviewers ask you to write LayerNorm or RMSNorm from memory. They also ask why language models do not use BatchNorm, which takes its averages across the other examples.
What you will learn
- Implement LayerNorm and RMSNorm forward passes
- Explain BatchNorm's train versus eval behavior
- Justify per-token normalization for variable-length sequences
Key terms
- feature
- one position in the row of numbers a layer outputs
- batch
- the inputs a model handles together
- RMSNorm
- a step that divides a row by its typical size
- root mean square (RMS)
- the square root of the average of the squares
- scale factor c
- a number that multiplies every value in a row
Prerequisites
Take Adam, AdamW and schedules first. It comes right before this lesson in the course.
Where this lesson picks up: You used warmup to keep the first training steps of a network safe. Networks that rescale the input of every layer need far less careful warmup, so what does that rescaling do?
What comes next
A row of zeros with no guard can make the loss NaN (not a number); the next lesson traces this and other broken training runs.
Next lesson: Initialization: He and Glorot from variance preservation
More lessons in this part
- SGD, momentum and the learning rate 15 min
- Adam, AdamW and schedules 15 min
- Initialization: He and Glorot from variance preservation 18 min
- Debugging training: NaNs, loss spikes, clipping and mixed precision 17 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.