Chapter 2: Deep learning

Normalization: BatchNorm, LayerNorm and RMSNorm

A normalization layer divides a row of numbers by its size, so every layer gets inputs on the same scale. Interviewers ask you to write LayerNorm or RMSNorm from memory. They also ask why language models do not use BatchNorm, which takes its averages across the other examples.

18 minInteractive lessonChapter 2: Deep learningLesson 5 of 18

What you will learn

  • Implement LayerNorm and RMSNorm forward passes
  • Explain BatchNorm's train versus eval behavior
  • Justify per-token normalization for variable-length sequences

Key terms

feature
one position in the row of numbers a layer outputs
batch
the inputs a model handles together
RMSNorm
a step that divides a row by its typical size
root mean square (RMS)
the square root of the average of the squares
scale factor c
a number that multiplies every value in a row

Prerequisites

Take Adam, AdamW and schedules first. It comes right before this lesson in the course.

Where this lesson picks up: You used warmup to keep the first training steps of a network safe. Networks that rescale the input of every layer need far less careful warmup, so what does that rescaling do?

What comes next

A row of zeros with no guard can make the loss NaN (not a number); the next lesson traces this and other broken training runs.

Next lesson: Initialization: He and Glorot from variance preservation

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.