PCA Variance with an Additional Feature

Reported in interviews at Citadel

Problem statement

You run PCA on a dataset with $P$ features and $N$ observations. The first principal component explains variance $\lambda_1$, the largest eigenvalue of the $P \times P$ covariance matrix.

Now you add one more feature to the dataset, making it $N \times (P+1)$, and rerun PCA.

Will the variance explained by the new first principal component be greater than, less than, or equal to $\lambda_1$? Prove your answer.

Show hint

How can the top eigenvalue of a covariance matrix be written as the solution of an optimisation problem over directions?

Relevant roles: ML Engineer, Data Scientist, Applied Scientist and Research Scientist.

Related free problems