Classic machine learning
Ridge and lasso
Like setting a shopping budget, regularization puts a cost on using more of something. Here that cost applies to coefficient size: ridge penalizes squared weights, while lasso penalizes absolute weights. Both can reduce sensitivity to noise at the cost of training fit, and lasso can set weights exactly to zero.
No penaltyRidge: z / 2Lasso: threshold at 1
How it works
- Standardize features using training data so the penalty treats comparable changes similarly.
- Add an L1 or L2 weight penalty to the loss, usually leaving the intercept unpenalized.
- Fit across penalty strengths using cross-validation with preprocessing inside each fold.
- Choose the penalty using validation performance and refit on the available training data.
$$\min_{b,w}\;\frac12\|y-b\mathbf1-Xw\|_2^2+\lambda R(w),\quad R(w)=\|w\|_1\text{ or }\tfrac12\|w\|_2^2$$
The knobs that matter
- lambda
- A larger penalty shrinks weights more and can underfit.
- L1 or L2
- L1 can select features; L2 tends to spread weight across correlated features.
- Feature scaling
- Scale determines how costly a coefficient is.
What interviewers ask
- Why does lasso produce exact zeros while ridge usually does not?
- Why can ridge have a unique solution when ordinary least squares does not?
- How do you choose penalty strength without leaking validation data?
- What happens when two features are highly correlated?
Common mistakes
- Penalizing features with different units without considering scaling.
- Assuming a feature dropped by lasso has no predictive value.
When to use it
Use ridge when overlapping or numerous features make estimates unstable, and lasso when sparse coefficients are useful. Neither fixes missing signal or data leakage.
Implement it (3 problems)
- Lasso Regression with ISTAProMediumSolve
Learn it
Interview questions about ridge and lasso
- Computing A Weighted LassoFreeMediumSolve