Chapter 1: Classical ML
Thresholds, precision and recall: the cut-off is a product decision
A score is not an action. You must choose a cut-off, and every cut-off makes two kinds of mistakes that cost different amounts. Interviewers ask where you would put the cut-off and which number you would report. They want a reason from the product, not 0.5 by default.
What you will learn
- Build a confusion matrix and compute precision, recall and F1
- Choose a threshold from the cost of false positives vs false negatives
- Argue for a metric given a product scenario
Key terms
- score
- A number from 0 to 1: the model's chance of spam.
- cut-off
- The score at or above which the filter flags an email; also called the threshold.
- precision
- Of the emails you flagged, the share that really were spam.
- recall
- Of the real spam emails, the share that you flagged.
Prerequisites
Take Training logistic regression: log loss, its gradient, and why not squared error first. It comes right before this lesson in the course.
Where this lesson picks up: You turned a line into a probability, so the model now gives each customer a score such as 0.3. The question you ended on was simple: the model says 0.3, so do we act?
What comes next
Every score so far came from customers the model learned from: would it hold up on new ones?
Next lesson: Honest splits and leakage: random, group and time-based
More lessons in this part
- Welcome to Classical ML: a model from start to finish 7 min
- Framing: what to predict, for whom, when, and what to beat 24 min
- Fitting a line: what 'best fit' means and how to read the slope 20 min
- The exact best line: two equations, and MAE 20 min
- From a line to a probability: the sigmoid 7 min
- Training logistic regression: log loss, its gradient, and why not squared error 7 min
- Honest splits and leakage: random, group and time-based 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.