Why Ridge Regression Is Well-Defined

Reported in interviews at DRW

Problem statement

Ordinary least squares solves $\hat{\beta} = (X^\top X)^{-1} X^\top y$, but when the columns of $X$ are collinear (or when there are more features than observations), $X^\top X$ is singular or badly ill-conditioned and the inverse blows up. Ridge regression instead solves $$\hat{\beta}_{\text{ridge}} = \arg\min_\beta \; \lVert y - X\beta \rVert_2^2 + \lambda \lVert \beta \rVert_2^2, \qquad \lambda > 0.$$

Prove rigorously why this works: show that adding $\lambda I$ makes the problem well-posed -- that $X^\top X + \lambda I$ is always invertible and the estimator is the unique minimizer -- and explain why this also fixes the ill-conditioning.

Show hint

What property of the Gram matrix follows from writing its quadratic form as a squared length?

Relevant roles: ML Engineer, Applied Scientist, Research Scientist and Data Scientist.

Related free problems