OLS Fitted Values with a Redundant Feature
Problem statement
You regress $y \in \mathbb{R}^n$ on the design matrix $\tilde{X} = [\mathbf{1},\, x_1,\, x_2]$, where $\mathbf{1}$ is the intercept column and $x_1, x_2$ are centered predictors ($\mathbf{1}^\top x_1 = \mathbf{1}^\top x_2 = 0$). Now define $z = x_1 - x_2$ and form the augmented matrix $X = [\mathbf{1},\, x_1,\, x_2,\, z]$.
- Prove whether the OLS fitted values $\hat{y}$ can change when you use $X$ instead of $\tilde{X}$.
- The augmented system $X\beta = \hat{y}$ has infinitely many solutions. Characterize the full set of OLS solutions $\hat{\beta}$ and give one explicit solution.
Show hint
Before touching coefficients, how would you describe $\hat y$ as a function of $y$ and the design matrix?
Relevant roles: ML Engineer, Applied Scientist, Research Scientist and Data Scientist.