Chapter 1: Classical ML
Preprocessing leakage and train-serve skew
Interviewers use these bugs to tell people who have shipped models from people who have only trained them. "Great in tests, bad with real users" is one of the most common debugging questions. In this lesson you find each bug, then fix it.
What you will learn
- Fit transforms inside cross-validation folds
- Explain out-of-fold target encoding
- Identify sources of training-serving skew and how logging serving features fixes it
Key terms
- pipeline
- the chain from raw data to a score
- fitted transform
- a step that learns numbers from the rows it is fit on
- feature selector
- keeps the features that rise and fall most with the labels
- accuracy
- the share of rows predicted right
- correlation r
- how a feature rises and falls with the labels, −1 to +1
Prerequisites
Take Features: scaling and missing values first. It comes right before this lesson in the course.
Where this lesson picks up: In the features lesson, you turned categories, scales and gaps into numbers with fitted encoders and scalers. Each one learns numbers from some rows, so which rows may it learn from?
What comes next
The linear toolkit is complete, but what if the pattern is not a line at all?
Next lesson: Decision trees: greedy splits
More lessons in this part
- Overfitting and the bias-variance decomposition 21 min
- Learning curves: is it bias or variance, and which fix? 7 min
- Cross-validation and choosing settings 7 min
- Regularization: ridge vs lasso and soft-thresholding 16 min
- Tuning the penalty: cross-validation and the one-standard-error rule 14 min
- Elastic net: when two features carry the same signal 14 min
- Features: turning categories into numbers 14 min
- Features: scaling and missing values 15 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.