Chapter 1: Classical ML

Preprocessing leakage and train-serve skew

Interviewers use these bugs to tell people who have shipped models from people who have only trained them. "Great in tests, bad with real users" is one of the most common debugging questions. In this lesson you find each bug, then fix it.

7 minInteractive lessonChapter 1: Classical MLLesson 10 of 21

What you will learn

  • Fit transforms inside cross-validation folds
  • Explain out-of-fold target encoding
  • Identify sources of training-serving skew and how logging serving features fixes it

Key terms

pipeline
the chain from raw data to a score
fitted transform
a step that learns numbers from the rows it is fit on
feature selector
keeps the features that rise and fall most with the labels
accuracy
the share of rows predicted right
correlation r
how a feature rises and falls with the labels, −1 to +1

Prerequisites

Take Features: scaling and missing values first. It comes right before this lesson in the course.

Where this lesson picks up: In the features lesson, you turned categories, scales and gaps into numbers with fitted encoders and scalers. Each one learns numbers from some rows, so which rows may it learn from?

What comes next

The linear toolkit is complete, but what if the pattern is not a line at all?

Next lesson: Decision trees: greedy splits

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.