Chapter 1: Classical ML

Cross-validation and choosing settings

A lucky split can make a weak setting look strong. In real work and in interviews, people trust a score only when the choice of setting never saw the rows behind it.

7 minInteractive lessonChapter 1: Classical MLLesson 10 of 25

What you will learn

  • Run k-fold and stratified k-fold and say what each fold estimates
  • Count training rows per fold
  • Explain why tuning and reporting on the same split is optimistic (nested CV)

Key terms

fold
one group of rows
check rows
the rows a model is scored on in one turn, the same as validation rows
cross-validation
each fold takes one turn as the check rows

Prerequisites

Take Learning curves: is it bias or variance, and which fix? first. It comes right before this lesson in the course.

Where this lesson picks up: You saw that one split is one roll of the dice: a different random split gives a different score. So how can you compare settings fairly?

What comes next

Now you can pick settings fairly; the next lesson looks at one setting that fights overfitting directly, a penalty on big weights.

Next lesson: Regularization: ridge vs lasso and soft-thresholding

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.