Chapter 1: Classical ML
Tuning the penalty: cross-validation and the one-standard-error rule
You will be able to pick $\lambda$ by cross-validation with the one-standard-error rule. Close scores can be luck, and interviewers ask how you would tune a penalty.
What you will learn
- Choose the penalty strength by CV with the one-standard-error rule
Key terms
- fold
- one part of the rows, held out once to test a model
- MSE
- mean squared error: the average of the squared prediction errors
- mean validation error
- the average error over the held-out folds
- SD
- standard deviation: the typical gap between a value and the mean
- SE
- standard error: how much a mean would wobble with fresh folds
Prerequisites
Take Regularization: ridge vs lasso and soft-thresholding first. It comes right before this lesson in the course.
Where this lesson picks up: Last lesson, ridge and lasso added a penalty whose strength is $\lambda$: a larger $\lambda$ shrinks the weights more. Two lessons ago, cross-validation scored settings on held-out folds. So how strong should $\lambda$ be?
What comes next
Next: what if two features carry the same signal? Lasso may keep just one of them; elastic net makes them share.
Next lesson: Elastic net: when two features carry the same signal
More lessons in this part
- Overfitting and the bias-variance decomposition 21 min
- Learning curves: is it bias or variance, and which fix? 7 min
- Cross-validation and choosing settings 7 min
- Regularization: ridge vs lasso and soft-thresholding 16 min
- Elastic net: when two features carry the same signal 14 min
- Features: turning categories into numbers 14 min
- Features: scaling and missing values 15 min
- Preprocessing leakage and train-serve skew 7 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.