Closed-Form Lasso Under Orthonormal Design
Problem statement
Suppose $X$ is an $n \times p$ design matrix with orthonormal columns ($X^T X = I$) and $y$ is the response vector.
- Show that the Lasso solution $\hat{\beta} = \arg\min_{\beta} \frac{1}{2}\|y - X\beta\|_2^2 + \lambda \|\beta\|_1$ reduces to componentwise soft-thresholding of $X^T y$. Give the exact formula for each $\hat{\beta}_j$.
- Explain how the threshold $\lambda$ controls feature selection: which coefficients are zeroed out and which survive?
- Compare the Lasso shrinkage to Ridge regression shrinkage under the same orthonormal design. How do the two operators differ in their treatment of small vs. large coefficients?
Show hint
What happens to the squared-error term when the columns of the design are orthonormal, and does the problem still couple the coordinates?
Relevant roles: ML Engineer, Applied Scientist, Research Scientist and Data Scientist.
Related free problems
- Computing A Weighted LassoClassical MLMedium