Chapter 1: Classical ML

Calibration and class imbalance

Many systems use the score as a number: a bid, an expected cost, a cut-off for a target precision. If the number is wrong, each decision that uses it is wrong too. Interviewers often ask how you check a score, how you fix it, and what happens when you remove most negatives before training.

20 minInteractive lessonChapter 1: Classical MLLesson 16 of 21

What you will learn

  • Measure calibration with expected calibration error
  • Recalibrate with Platt scaling
  • Correct predicted probabilities after downsampling the majority class

Key terms

calibrated
scores match real rates: of customers scored 0.8, about 80% churn
ECE
A single number for the gap between scores and real rates.
Platt scaling
A small fit that maps a model score to a better probability.
Odds
The chance of a positive divided by the chance of a negative.
log-odds
the log of the odds; a logistic model's score z is exactly this

Prerequisites

Take ROC, PR curves and AUC first. It comes right before this lesson in the course.

Where this lesson picks up: You used ROC AUC to check how well a model puts positives above negatives. That left one question open: when a model says 0.8, is the chance really 80%?

What comes next

A calibrated probability is one number, so next you ask how wide the range is where the truth will land.

Next lesson: Prediction intervals and conformal prediction

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.