Chapter 2: Deep learning
Softmax, cross-entropy and the stable log-sum-exp
Most neural-network classifiers turn their scores into probabilities, then into one loss number to learn from. Interviewers ask you to derive the gradient of that loss on a whiteboard. Then they ask why your code does not print inf on a score of 1000. This lesson answers both, with one cat photo and small numbers.
What you will learn
- Derive the gradient of softmax cross-entropy with respect to logits
- Implement a numerically stable log-softmax
- Choose a loss for binary, multi-class and imbalanced (focal) outputs
Key terms
- e
- a fixed number, about 2.718
- ln
- the log with base e; it undoes e
- logit
- a raw score from the last layer, one per class
- loss
- one number that says how wrong the model is
- overflow
- a number too big for the computer to store
Prerequisites
Take Forward pass: layers, activations and shapes first. It comes right before this lesson in the course.
Where this lesson picks up: You traced a forward pass to its end: a logits grid of shape (N, C), with one raw score per class for each example. These scores are not probabilities yet, so how do you turn them into a loss and a gradient?
What comes next
You have p − y at the logits, so how do you push it back through the last layer to get the weight gradients?
Next lesson: Backprop by hand on a two-layer MLP
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.