Closed-Form Lasso Under Orthonormal Design

Reported in interviews at Citadel Hudson River Trading Jane Street Two Sigma

Problem statement

Suppose $X$ is an $n \times p$ design matrix with orthonormal columns ($X^T X = I$) and $y$ is the response vector.

  1. Show that the Lasso solution $\hat{\beta} = \arg\min_{\beta} \frac{1}{2}\|y - X\beta\|_2^2 + \lambda \|\beta\|_1$ reduces to componentwise soft-thresholding of $X^T y$. Give the exact formula for each $\hat{\beta}_j$.
  1. Explain how the threshold $\lambda$ controls feature selection: which coefficients are zeroed out and which survive?
  1. Compare the Lasso shrinkage to Ridge regression shrinkage under the same orthonormal design. How do the two operators differ in their treatment of small vs. large coefficients?
Show hint

What happens to the squared-error term when the columns of the design are orthonormal, and does the problem still couple the coordinates?

Relevant roles: ML Engineer, Applied Scientist, Research Scientist and Data Scientist.

Related free problems