Classic machine learning

Gradient boosting

Like revising an essay after each round of feedback, gradient boosting improves a prediction in small rounds. Each new tree fits a direction that reduces the current loss. The model adds a scaled version of that tree to the predictions so far.

Current prediction5; target = 8Fit residualTree output = 8 - 5 = 3Scale tree output0.5 x 3 = 1.5Add update5 + 1.5 = 6.5
For half squared error, the next tree fits residuals; a learning rate of 0.5 halves this example's error.

How it works

  1. Start with a constant prediction chosen to minimize the training loss.
  2. Compute negative loss gradients at current predictions, which are residuals for half squared error.
  3. Fit a small tree to those values and choose leaf updates for the loss.
  4. Add its predictions multiplied by a learning rate.
  5. Repeat while validation performance improves, using early stopping to select the number of rounds.

$$F_m(x)=F_{m-1}(x)+\eta h_m(x)$$

The knobs that matter

learning_rate
Smaller updates usually require more trees.
n_estimators
Too many rounds can fit noise; select using validation.
max_depth
Deeper trees capture more complex interactions and can overfit.
subsample
Training each tree on fewer rows adds randomness.

What interviewers ask

  • How does gradient boosting differ from bagging?
  • What does each tree learn for squared error versus log loss?
  • How do learning rate and number of trees interact?
  • Why use shallow trees and early stopping?

Common mistakes

  • Saying every boosting loss trains trees directly on raw residuals.
  • Assuming boosting cannot overfit because each tree is small.

When to use it

Use it for tabular prediction where nonlinear patterns and interactions matter. Sequential training and tuning can make it less convenient under tight training budgets.

Implement it (4 problems)

Learn it

Interview questions about gradient boosting