Chapter 1: Classical ML

Honest splits and leakage: random, group and time-based

A model can score very well on its test rows and then fail after launch. A common cause is a leak: training saw information that the model will not have when it predicts. Interviewers ask about leaks because finding them is a core skill in real work.

18 minInteractive lessonChapter 1: Classical MLLesson 5 of 21

What you will learn

  • Pick random, group or time-based splits for a dataset
  • Diagnose target, look-ahead and duplicate leakage, using the test: would this value exist at prediction time?
  • Apply 'split first, then everything inside the training fold'

Key terms

leak
Training saw information the model will not have when it predicts.
group split
All rows of one patient or customer stay on one side.
time split
Train only on days before the test days.
embargo
Days dropped before each test block, banned from training.
target leakage
A feature that holds the answer.

Prerequisites

Take Thresholds, precision and recall: the cut-off is a product decision first. It comes right before this lesson in the course.

Where this lesson picks up: You chose a cut-off and used precision and recall on test rows to price two kinds of mistakes. That left a question open: would those scores stay as high on new customers?

What comes next

Your honest score is lower than your training score, so the next lesson asks why, and what you change.

Next lesson: Overfitting and the bias-variance decomposition

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.