Classic machine learning

PCA

Like finding the best camera angle to show a spread of objects, PCA chooses directions that reveal the most variation. It projects centered features onto perpendicular axes called principal components. Keeping only a few components compresses the data without using target labels.

-3-2-10123-3-2-10123Centered feature 1Centered feature 2
Centered observationsFirst component direction
All four points lie along (1, 1), so projection onto its unit direction preserves all variance.

How it works

  1. Center features using training means and scale first if units should have comparable influence.
  2. Compute a singular value decomposition of the centered matrix or eigenvectors of its covariance matrix.
  3. Order component directions by decreasing explained variance.
  4. Project onto chosen directions and apply the same fitted transformation to new data.
  5. Choose component count using reconstruction needs or validation of the downstream task.

$$Z=(X-\mathbf1\mu^T)V_k,\quad V_k^TV_k=I$$

The knobs that matter

n_components
More components retain more training variance but compress less.
Feature scaling
Changes which directions contain the most variance.
Whitening
Rescales retained components to unit variance and can amplify low-variance noise.

What interviewers ask

  • How do covariance eigenvalues relate to explained variance?
  • Why center the data, and when should you standardize it?
  • How would you derive PCA from a reconstruction-error objective?
  • Can a component with low variance still matter for prediction?

Common mistakes

  • Fitting PCA before splitting data or outside cross-validation folds.
  • Assuming directions with most variance must be the most predictive.
  • Interpreting perpendicular components as necessarily statistically independent.

When to use it

Use it to compress correlated numeric features, visualize data, or remove some low-variance variation. Avoid discarding components blindly when small variations may contain the target signal.

Implement it (6 problems)

Learn it

Interview questions about pca