Chapter 1: Classical ML
Cross-validation and choosing settings
A lucky split can make a weak setting look strong. In real work and in interviews, people trust a score only when the choice of setting never saw the rows behind it.
What you will learn
- Run k-fold and stratified k-fold and say what each fold estimates
- Count training rows per fold
- Explain why tuning and reporting on the same split is optimistic (nested CV)
Key terms
- fold
- one group of rows
- check rows
- the rows a model is scored on in one turn, the same as validation rows
- cross-validation
- each fold takes one turn as the check rows
Prerequisites
Take Learning curves: is it bias or variance, and which fix? first. It comes right before this lesson in the course.
Where this lesson picks up: You saw that one split is one roll of the dice: a different random split gives a different score. So how can you compare settings fairly?
What comes next
Now you can pick settings fairly; the next lesson looks at one setting that fights overfitting directly, a penalty on big weights.
Next lesson: Regularization: ridge vs lasso and soft-thresholding
More lessons in this part
- Overfitting and the bias-variance decomposition 21 min
- Learning curves: is it bias or variance, and which fix? 7 min
- Regularization: ridge vs lasso and soft-thresholding 16 min
- Tuning the penalty: cross-validation and the one-standard-error rule 14 min
- Elastic net: when two features carry the same signal 14 min
- Features: turning categories into numbers 14 min
- Features: scaling and missing values 15 min
- Preprocessing leakage and train-serve skew 7 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.