Chapter 1: Classical ML

Regularization: ridge vs lasso and soft-thresholding

A model can fit training data well but predict new data poorly. Weight penalties can help, and interviews ask why their effects differ.

16 minInteractive lessonChapter 1: Classical MLLesson 11 of 25

What you will learn

  • Explain the L1 versus L2 geometry and soft-thresholding

Key terms

predictor
an input column the model uses, such as the house area
penalty
an extra cost added to the fit cost for large weights
lambda $\lambda$
the penalty strength; $\lambda = 0$ adds no penalty
ridge
a penalty on squared weights, called L2
lasso
a penalty on absolute weights, called L1

Prerequisites

Take Cross-validation and choosing settings first. It comes right before this lesson in the course.

Where this lesson picks up: You used held-out folds to compare settings. What can a setting change inside a model?

What comes next

How should you choose penalty strength when predictors share a signal?

Next lesson: Tuning the penalty: cross-validation and the one-standard-error rule

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.