Chapter 1: Classical ML
Regularization: ridge vs lasso and soft-thresholding
A model can fit training data well but predict new data poorly. Weight penalties can help, and interviews ask why their effects differ.
What you will learn
- Explain the L1 versus L2 geometry and soft-thresholding
Key terms
- predictor
- an input column the model uses, such as the house area
- penalty
- an extra cost added to the fit cost for large weights
- lambda $\lambda$
- the penalty strength; $\lambda = 0$ adds no penalty
- ridge
- a penalty on squared weights, called L2
- lasso
- a penalty on absolute weights, called L1
Prerequisites
Take Cross-validation and choosing settings first. It comes right before this lesson in the course.
Where this lesson picks up: You used held-out folds to compare settings. What can a setting change inside a model?
What comes next
How should you choose penalty strength when predictors share a signal?
Next lesson: Tuning the penalty: cross-validation and the one-standard-error rule
More lessons in this part
- Overfitting and the bias-variance decomposition 21 min
- Learning curves: is it bias or variance, and which fix? 7 min
- Cross-validation and choosing settings 7 min
- Tuning the penalty: cross-validation and the one-standard-error rule 14 min
- Elastic net: when two features carry the same signal 14 min
- Features: turning categories into numbers 14 min
- Features: scaling and missing values 15 min
- Preprocessing leakage and train-serve skew 7 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.