Chapter 2: Deep learning
Backprop by hand on a two-layer MLP
Training needs the slope of the loss for every weight and bias. Backprop gets them all in one sweep back through the network, and nudges prove the result. Interviewers ask you to do both by hand on a small network.
What you will learn
- Derive weight and bias gradients for a two-layer MLP
- Verify gradients with a finite-difference check
- Explain vanishing and exploding gradients and dead ReLUs
Key terms
- slope
- How far an output moves, divided by the nudge of its input.
- nudge
- A small change to an input, such as 0.01.
- chain rule
- Multiply the slopes along a path of steps.
- ReLU
- Keeps a positive number and turns a negative one into 0.
- backprop
- One sweep back through the network for every weight's slope.
Prerequisites
Take Softmax, cross-entropy and the stable log-sum-exp first. It comes right before this lesson in the course.
Where this lesson picks up: In the softmax lesson, you found that softmax with cross-entropy gives the logits the simple slope $p - y$. But the weights sit further back in the network, so how do you get their slopes?
What comes next
You now have every gradient. How big a step should each weight take?
Next lesson: SGD, momentum and the learning rate
More lessons in this part
- Forward pass: layers, activations and shapes 20 min
- Softmax, cross-entropy and the stable log-sum-exp 19 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.