Chapter 2: Deep learning

Backprop by hand on a two-layer MLP

Training needs the slope of the loss for every weight and bias. Backprop gets them all in one sweep back through the network, and nudges prove the result. Interviewers ask you to do both by hand on a small network.

18 minInteractive lessonChapter 2: Deep learningLesson 3 of 18

What you will learn

  • Derive weight and bias gradients for a two-layer MLP
  • Verify gradients with a finite-difference check
  • Explain vanishing and exploding gradients and dead ReLUs

Key terms

slope
How far an output moves, divided by the nudge of its input.
nudge
A small change to an input, such as 0.01.
chain rule
Multiply the slopes along a path of steps.
ReLU
Keeps a positive number and turns a negative one into 0.
backprop
One sweep back through the network for every weight's slope.

Prerequisites

Take Softmax, cross-entropy and the stable log-sum-exp first. It comes right before this lesson in the course.

Where this lesson picks up: In the softmax lesson, you found that softmax with cross-entropy gives the logits the simple slope $p - y$. But the weights sit further back in the network, so how do you get their slopes?

What comes next

You now have every gradient. How big a step should each weight take?

Next lesson: SGD, momentum and the learning rate

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.