Chapter 1: Classical ML
ROC, PR curves and AUC
You will be able to draw one curve that shows every threshold, find its area by hand, and say what that area means. You will also read a second curve that counts false alarms. Interviewers often ask for this area by hand, and for the right curve in rare cases such as fraud.
What you will learn
- Trace an ROC curve and compute AUC
- Interpret AUC as a pairwise ranking probability
- Read a precision-recall curve
Key terms
- pair
- one customer who left (a positive) and one who stayed (a negative)
- threshold
- you flag every customer whose score is at or above it
- TPR
- true positive rate: another name for recall, flagged positives divided by all positives
- FPR
- false positive rate: flagged negatives divided by all negatives
- AUC
- the area under the ROC curve; one number for all thresholds
Prerequisites
Take Support vector machines: margins and hinge loss first. It comes right before this lesson in the course.
Where this lesson picks up: Earlier you picked one threshold, counted TP, FP, TN and FN, and found precision and recall. But each threshold gives a different confusion matrix, so how do you judge the scores at every threshold at once?
What comes next
Next: AUC only uses the order of the scores, so is a score of 0.8 really an 80% chance of leaving?
Next lesson: Calibration and class imbalance
More lessons in this part
- Calibration and class imbalance 20 min
- Prediction intervals and conformal prediction 18 min
- Explaining a model: permutation importance and SHAP 17 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.