Classic machine learning
PCA
Like finding the best camera angle to show a spread of objects, PCA chooses directions that reveal the most variation. It projects centered features onto perpendicular axes called principal components. Keeping only a few components compresses the data without using target labels.
Centered observationsFirst component direction
How it works
- Center features using training means and scale first if units should have comparable influence.
- Compute a singular value decomposition of the centered matrix or eigenvectors of its covariance matrix.
- Order component directions by decreasing explained variance.
- Project onto chosen directions and apply the same fitted transformation to new data.
- Choose component count using reconstruction needs or validation of the downstream task.
$$Z=(X-\mathbf1\mu^T)V_k,\quad V_k^TV_k=I$$
The knobs that matter
- n_components
- More components retain more training variance but compress less.
- Feature scaling
- Changes which directions contain the most variance.
- Whitening
- Rescales retained components to unit variance and can amplify low-variance noise.
What interviewers ask
- How do covariance eigenvalues relate to explained variance?
- Why center the data, and when should you standardize it?
- How would you derive PCA from a reconstruction-error objective?
- Can a component with low variance still matter for prediction?
Common mistakes
- Fitting PCA before splitting data or outside cross-validation folds.
- Assuming directions with most variance must be the most predictive.
- Interpreting perpendicular components as necessarily statistically independent.
When to use it
Use it to compress correlated numeric features, visualize data, or remove some low-variance variation. Avoid discarding components blindly when small variations may contain the target signal.
Implement it (6 problems)
- PCA Reconstruction ErrorProMediumSolve
- PCA Color AugmentationProHardSolve