OLS Fitted Values with a Redundant Feature

Reported in interviews at Tower Research Capital

Problem statement

You regress $y \in \mathbb{R}^n$ on the design matrix $\tilde{X} = [\mathbf{1},\, x_1,\, x_2]$, where $\mathbf{1}$ is the intercept column and $x_1, x_2$ are centered predictors ($\mathbf{1}^\top x_1 = \mathbf{1}^\top x_2 = 0$). Now define $z = x_1 - x_2$ and form the augmented matrix $X = [\mathbf{1},\, x_1,\, x_2,\, z]$.

  1. Prove whether the OLS fitted values $\hat{y}$ can change when you use $X$ instead of $\tilde{X}$.
  1. The augmented system $X\beta = \hat{y}$ has infinitely many solutions. Characterize the full set of OLS solutions $\hat{\beta}$ and give one explicit solution.
Show hint

Before touching coefficients, how would you describe $\hat y$ as a function of $y$ and the design matrix?

Relevant roles: ML Engineer, Applied Scientist, Research Scientist and Data Scientist.

Related free problems