Classic machine learning
Linear regression
Like drawing a straight trend through a scatter of test scores, linear regression finds a simple pattern in noisy data. It predicts a number by adding weighted features and an intercept. Ordinary least squares chooses weights that minimize squared prediction errors.
Least-squares lineObserved scores
How it works
- Build a feature matrix and include an intercept if the baseline need not be zero.
- Predict each target as a weighted sum of its features.
- Minimize squared residuals using a matrix solver or gradient descent.
- Evaluate on held-out data and inspect residuals for patterns.
$$\hat y=b+\sum_{j=1}^{p}w_jx_j$$
The knobs that matter
- Feature choice
- Transformations and interactions let a linear model describe more patterns.
- Intercept
- Include it unless a zero baseline is justified.
- Regularization
- A penalty can stabilize coefficients when features overlap.
What interviewers ask
- When is the least-squares solution unique, and what can you do if the feature matrix is rank deficient?
- What assumptions do you need for estimation and for statistical inference?
- How do you interpret coefficients when features are correlated?
- Why do residuals sum to zero when you fit an intercept?
Common mistakes
- Treating a coefficient as causal without addressing confounding.
- Explicitly inverting the normal-equation matrix even when it is nearly singular.
When to use it
Use it for a readable baseline when a weighted sum describes the target reasonably well. Add suitable features or choose another model when residuals show strong nonlinear structure.
Implement it (6 problems)
Build it step by step
- NumPy House Price Regression24 steps, your code runs in the browser
- NumPy Multiple Linear Regression GD28 steps, your code runs in the browser
Learn it
Interview questions about linear regression
- Omitted Variable Bias in OLSFreeMediumSolve