Angle Between y and y-hat in Ridge Regression
Problem statement
In a linear regression model $y = X\beta + \epsilon$, consider the ridge regression predicted vector
$$\hat{y}_{\text{ridge}} = X(X^\top X + \lambda I)^{-1}X^\top y$$
for a regularization parameter $\lambda > 0$.
Prove or disprove the following claim: for any choice of $\lambda > 0$, the angle between the observation vector $y$ and the predicted vector $\hat{y}_{\text{ridge}}$ is the same (i.e., the angle is independent of $\lambda$).
Show hint
What geometric property makes the OLS fitted vector special, and does the ridge smoother matrix share it?
Relevant roles: ML Engineer, Applied Scientist, Research Scientist and Data Scientist.