Chapter 1: Classical ML

ROC, PR curves and AUC

You will be able to draw one curve that shows every threshold, find its area by hand, and say what that area means. You will also read a second curve that counts false alarms. Interviewers often ask for this area by hand, and for the right curve in rare cases such as fraud.

15 minInteractive lessonChapter 1: Classical MLLesson 20 of 26

What you will learn

  • Trace an ROC curve and compute AUC
  • Interpret AUC as a pairwise ranking probability
  • Read a precision-recall curve

Key terms

pair
one customer who left (a positive) and one who stayed (a negative)
threshold
you flag every customer whose score is at or above it
TPR
true positive rate: another name for recall, flagged positives divided by all positives
FPR
false positive rate: flagged negatives divided by all negatives
AUC
the area under the ROC curve; one number for all thresholds

Prerequisites

Take Support vector machines: margins and hinge loss first. It comes right before this lesson in the course.

Where this lesson picks up: Earlier you picked one threshold, counted TP, FP, TN and FN, and found precision and recall. But each threshold gives a different confusion matrix, so how do you judge the scores at every threshold at once?

What comes next

Next: AUC only uses the order of the scores, so is a score of 0.8 really an 80% chance of leaving?

Next lesson: Calibration and class imbalance

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.