Chapter 1: Classical ML
Training logistic regression: log loss, its gradient, and why not squared error
A model can learn only with a score for each guess and a direction for each weight. You will be able to score a predicted chance and find which way each weight must move. You will also see why squared error trains this model badly, a common interview question.
What you will learn
- Compute log loss and the gradient (p - y) x for one example
- Explain why squared error is a poor loss for a sigmoid output
Key terms
- log loss
- a score for one guess: minus ln of the chance it gave to what happened
- ln
- the natural log, the log with base e
- gradient
- the slope of the loss for a weight, which shows which way to move it
- likelihood
- the chance the model gave to all the outcomes that really happened
Prerequisites
Take From a line to a probability: the sigmoid first. It comes right before this lesson in the course.
Where this lesson picks up: Last time, the sigmoid turned a line's score into a chance between 0 and 1. The model now says 0.7 for a customer who stayed, so how do we score that guess, and which way should the weight move?
What comes next
The model says 0.3 for a customer: do we act?
Next lesson: Thresholds, precision and recall: the cut-off is a product decision
More lessons in this part
- Welcome to Classical ML: a model from start to finish 7 min
- Framing: what to predict, for whom, when, and what to beat 24 min
- Fitting a line: what 'best fit' means and how to read the slope 20 min
- The exact best line: two equations, and MAE 20 min
- From a line to a probability: the sigmoid 7 min
- Thresholds, precision and recall: the cut-off is a product decision 16 min
- Honest splits and leakage: random, group and time-based 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.