Chapter 2: Deep learning

Softmax, cross-entropy and the stable log-sum-exp

Most neural-network classifiers turn their scores into probabilities, then into one loss number to learn from. Interviewers ask you to derive the gradient of that loss on a whiteboard. Then they ask why your code does not print inf on a score of 1000. This lesson answers both, with one cat photo and small numbers.

19 minInteractive lessonChapter 2: Deep learningLesson 2 of 18

What you will learn

  • Derive the gradient of softmax cross-entropy with respect to logits
  • Implement a numerically stable log-softmax
  • Choose a loss for binary, multi-class and imbalanced (focal) outputs

Key terms

e
a fixed number, about 2.718
ln
the log with base e; it undoes e
logit
a raw score from the last layer, one per class
loss
one number that says how wrong the model is
overflow
a number too big for the computer to store

Prerequisites

Take Forward pass: layers, activations and shapes first. It comes right before this lesson in the course.

Where this lesson picks up: You traced a forward pass to its end: a logits grid of shape (N, C), with one raw score per class for each example. These scores are not probabilities yet, so how do you turn them into a loss and a gradient?

What comes next

You have p − y at the logits, so how do you push it back through the last layer to get the weight gradients?

Next lesson: Backprop by hand on a two-layer MLP

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.