Chapter 1: Classical ML

Thresholds, precision and recall: the cut-off is a product decision

A score is not an action. You must choose a cut-off, and every cut-off makes two kinds of mistakes that cost different amounts. Interviewers ask where you would put the cut-off and which number you would report. They want a reason from the product, not 0.5 by default.

16 minInteractive lessonChapter 1: Classical MLLesson 4 of 21

What you will learn

  • Build a confusion matrix and compute precision, recall and F1
  • Choose a threshold from the cost of false positives vs false negatives
  • Argue for a metric given a product scenario

Key terms

score
A number from 0 to 1: the model's chance of spam.
cut-off
The score at or above which the filter flags an email; also called the threshold.
precision
Of the emails you flagged, the share that really were spam.
recall
Of the real spam emails, the share that you flagged.

Prerequisites

Take Training logistic regression: log loss, its gradient, and why not squared error first. It comes right before this lesson in the course.

Where this lesson picks up: You turned a line into a probability, so the model now gives each customer a score such as 0.3. The question you ended on was simple: the model says 0.3, so do we act?

What comes next

Every score so far came from customers the model learned from: would it hold up on new ones?

Next lesson: Honest splits and leakage: random, group and time-based

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.