Principal Component Analysis (PCA) from Scratch

MediumML Coding

Problem statement

Write a function that performs Principal Component Analysis (PCA) from scratch. The input is a 2D NumPy array where each row is a sample and each column is a feature, plus an integer k. The function should:

  1. Standardize each feature (subtract the mean, divide by the population standard deviation).
  2. Compute the covariance matrix of the standardized data.
  3. Find its eigenvalues and eigenvectors.
  4. Return the k eigenvectors with the largest eigenvalues as the columns of an array of shape (n_features, k), rounded to 4 decimals.

Sign convention: an eigenvector v and -v are equally valid. For each returned eigenvector, find its first element with absolute value greater than 1e-10; if that element is negative, multiply the whole vector by -1.

Note: use np.linalg.eigh for the eigendecomposition, since covariance matrices are symmetric.

Example

Input:

data = np.array([[1, 2], [3, 4], [5, 6]])
k = 1

Output:

[[0.7071], [0.7071]]

Reasoning: the points lie on the line y = x + 1. After standardization both features are identical, so the direction of maximum variance is $[1, 1] / \sqrt{2} = [0.7071, 0.7071]$, which captures all of the variance.

Show hint

Which quantity measures how much of the data's spread lies along a given direction, and which matrix encodes it for every direction at once?

Relevant roles: ML Engineer, Data Scientist, Applied Scientist and Research Scientist.

Related free problems