Classic machine learning

Linear regression

Like drawing a straight trend through a scatter of test scores, linear regression finds a simple pattern in noisy data. It predicts a number by adding weighted features and an intercept. Ordinary least squares chooses weights that minimize squared prediction errors.

012340246810Hours studiedPractice score
Least-squares lineObserved scores
The line predicts 3, 5 and 7 for scores 2, 7 and 6: misses of -1, +2 and -1 sum to zero, and their squares sum to 6.

How it works

  1. Build a feature matrix and include an intercept if the baseline need not be zero.
  2. Predict each target as a weighted sum of its features.
  3. Minimize squared residuals using a matrix solver or gradient descent.
  4. Evaluate on held-out data and inspect residuals for patterns.

$$\hat y=b+\sum_{j=1}^{p}w_jx_j$$

The knobs that matter

Feature choice
Transformations and interactions let a linear model describe more patterns.
Intercept
Include it unless a zero baseline is justified.
Regularization
A penalty can stabilize coefficients when features overlap.

What interviewers ask

  • When is the least-squares solution unique, and what can you do if the feature matrix is rank deficient?
  • What assumptions do you need for estimation and for statistical inference?
  • How do you interpret coefficients when features are correlated?
  • Why do residuals sum to zero when you fit an intercept?

Common mistakes

  • Treating a coefficient as causal without addressing confounding.
  • Explicitly inverting the normal-equation matrix even when it is nearly singular.

When to use it

Use it for a readable baseline when a weighted sum describes the target reasonably well. Add suitable features or choose another model when residuals show strong nonlinear structure.

Implement it (6 problems)

Build it step by step

Learn it

Interview questions about linear regression