Chapter 2: Deep learning

Adam, AdamW and schedules

Use gradient history to scale each weight, repair averages that start at zero, and combine weight decay with a changing learning rate.

15 minInteractive lessonChapter 2: Deep learningLesson 4b

What you will learn

  • Write one AdamW update including bias correction
  • Explain why decoupled weight decay differs from an L2 penalty under Adam
  • Implement a warmup-plus-cosine learning-rate schedule

Key terms

Adam
an optimizer that scales each weight using its own gradient history
AdamW
Adam with a separate weight shrink
schedule
a rule that changes the learning rate over training steps

Prerequisites

Take SGD, momentum and the learning rate first. It comes right before this lesson in the course.

Where this lesson picks up: SGD scales gradients by a learning rate; momentum smooths them with a running average.

What comes next

Next, normalization controls the scale of values inside a layer.

Next lesson: Normalization: BatchNorm, LayerNorm and RMSNorm

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.