Chapter 1: Classical ML
Calibration and class imbalance
Many systems use the score as a number: a bid, an expected cost, a cut-off for a target precision. If the number is wrong, each decision that uses it is wrong too. Interviewers often ask how you check a score, how you fix it, and what happens when you remove most negatives before training.
What you will learn
- Measure calibration with expected calibration error
- Recalibrate with Platt scaling
- Correct predicted probabilities after downsampling the majority class
Key terms
- calibrated
- scores match real rates: of customers scored 0.8, about 80% churn
- ECE
- A single number for the gap between scores and real rates.
- Platt scaling
- A small fit that maps a model score to a better probability.
- Odds
- The chance of a positive divided by the chance of a negative.
- log-odds
- the log of the odds; a logistic model's score z is exactly this
Prerequisites
Take ROC, PR curves and AUC first. It comes right before this lesson in the course.
Where this lesson picks up: You used ROC AUC to check how well a model puts positives above negatives. That left one question open: when a model says 0.8, is the chance really 80%?
What comes next
A calibrated probability is one number, so next you ask how wide the range is where the truth will land.
Next lesson: Prediction intervals and conformal prediction
More lessons in this part
- ROC, PR curves and AUC 15 min
- Prediction intervals and conformal prediction 18 min
- Explaining a model: permutation importance and SHAP 17 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.