Variance Blowup Under Severe Multicollinearity
Problem statement
You are running an OLS regression of $Y$ on two standardized features $X_1$ and $X_2$ (with an intercept), and you discover that $\text{Corr}(X_1, X_2) = \rho = 0.999$.
- Write out the $(X^\top X)^{-1}$ matrix for this setup (using the fact that the features are standardized). Express $\text{Var}(\hat{\beta})$ in terms of $(X^\top X)^{-1}$ and $\sigma^2$, then quantify how the diagonal entries blow up as $\rho \to 1$.
- Explain how ridge regression and lasso each address this problem. For ridge with penalty parameter $\lambda$, write the closed-form estimator, show how adding $\lambda$ changes the effective condition number of $X^\top X$, and explain the bias-variance trade-off you are making. For lasso, describe qualitatively how it handles near-collinear features differently from ridge.
Show hint
What does $X^ op X$ reduce to when the feature columns are standardized, and which familiar matrix is it a multiple of?
Relevant roles: Research Scientist, Applied Scientist, Data Scientist and ML Engineer.