Angle Between y and y-hat in Ridge Regression

Reported in interviews at Citadel PDT Partners

Problem statement

In a linear regression model $y = X\beta + \epsilon$, consider the ridge regression predicted vector

$$\hat{y}_{\text{ridge}} = X(X^\top X + \lambda I)^{-1}X^\top y$$

for a regularization parameter $\lambda > 0$.

Prove or disprove the following claim: for any choice of $\lambda > 0$, the angle between the observation vector $y$ and the predicted vector $\hat{y}_{\text{ridge}}$ is the same (i.e., the angle is independent of $\lambda$).

Show hint

What geometric property makes the OLS fitted vector special, and does the ridge smoother matrix share it?

Relevant roles: ML Engineer, Applied Scientist, Research Scientist and Data Scientist.

Related free problems