Principal Component Analysis (PCA) from Scratch
Problem statement
Write a function that performs Principal Component Analysis (PCA) from scratch. The input is a 2D NumPy array where each row is a sample and each column is a feature, plus an integer k. The function should:
- Standardize each feature (subtract the mean, divide by the population standard deviation).
- Compute the covariance matrix of the standardized data.
- Find its eigenvalues and eigenvectors.
- Return the
keigenvectors with the largest eigenvalues as the columns of an array of shape(n_features, k), rounded to 4 decimals.
Sign convention: an eigenvector v and -v are equally valid. For each returned eigenvector, find its first element with absolute value greater than 1e-10; if that element is negative, multiply the whole vector by -1.
Note: use np.linalg.eigh for the eigendecomposition, since covariance matrices are symmetric.
Example
Input:
data = np.array([[1, 2], [3, 4], [5, 6]])
k = 1
Output:
[[0.7071], [0.7071]]
Reasoning: the points lie on the line y = x + 1. After standardization both features are identical, so the direction of maximum variance is $[1, 1] / \sqrt{2} = [0.7071, 0.7071]$, which captures all of the variance.
Show hint
Which quantity measures how much of the data's spread lies along a given direction, and which matrix encodes it for every direction at once?
Relevant roles: ML Engineer, Data Scientist, Applied Scientist and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium