PCA Variance with an Additional Feature
Problem statement
You run PCA on a dataset with $P$ features and $N$ observations. The first principal component explains variance $\lambda_1$, the largest eigenvalue of the $P \times P$ covariance matrix.
Now you add one more feature to the dataset, making it $N \times (P+1)$, and rerun PCA.
Will the variance explained by the new first principal component be greater than, less than, or equal to $\lambda_1$? Prove your answer.
Show hint
How can the top eigenvalue of a covariance matrix be written as the solution of an optimisation problem over directions?
Relevant roles: ML Engineer, Data Scientist, Applied Scientist and Research Scientist.