Chapter 1: Classical ML
Honest splits and leakage: random, group and time-based
A model can score very well on its test rows and then fail after launch. A common cause is a leak: training saw information that the model will not have when it predicts. Interviewers ask about leaks because finding them is a core skill in real work.
What you will learn
- Pick random, group or time-based splits for a dataset
- Diagnose target, look-ahead and duplicate leakage, using the test: would this value exist at prediction time?
- Apply 'split first, then everything inside the training fold'
Key terms
- leak
- Training saw information the model will not have when it predicts.
- group split
- All rows of one patient or customer stay on one side.
- time split
- Train only on days before the test days.
- embargo
- Days dropped before each test block, banned from training.
- target leakage
- A feature that holds the answer.
Prerequisites
Take Thresholds, precision and recall: the cut-off is a product decision first. It comes right before this lesson in the course.
Where this lesson picks up: You chose a cut-off and used precision and recall on test rows to price two kinds of mistakes. That left a question open: would those scores stay as high on new customers?
What comes next
Your honest score is lower than your training score, so the next lesson asks why, and what you change.
Next lesson: Overfitting and the bias-variance decomposition
More lessons in this part
- Welcome to Classical ML: a model from start to finish 7 min
- Framing: what to predict, for whom, when, and what to beat 24 min
- Fitting a line: what 'best fit' means and how to read the slope 20 min
- The exact best line: two equations, and MAE 20 min
- From a line to a probability: the sigmoid 7 min
- Training logistic regression: log loss, its gradient, and why not squared error 7 min
- Thresholds, precision and recall: the cut-off is a product decision 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.