Classic machine learning
Gradient boosting
Like revising an essay after each round of feedback, gradient boosting improves a prediction in small rounds. Each new tree fits a direction that reduces the current loss. The model adds a scaled version of that tree to the predictions so far.
How it works
- Start with a constant prediction chosen to minimize the training loss.
- Compute negative loss gradients at current predictions, which are residuals for half squared error.
- Fit a small tree to those values and choose leaf updates for the loss.
- Add its predictions multiplied by a learning rate.
- Repeat while validation performance improves, using early stopping to select the number of rounds.
$$F_m(x)=F_{m-1}(x)+\eta h_m(x)$$
The knobs that matter
- learning_rate
- Smaller updates usually require more trees.
- n_estimators
- Too many rounds can fit noise; select using validation.
- max_depth
- Deeper trees capture more complex interactions and can overfit.
- subsample
- Training each tree on fewer rows adds randomness.
What interviewers ask
- How does gradient boosting differ from bagging?
- What does each tree learn for squared error versus log loss?
- How do learning rate and number of trees interact?
- Why use shallow trees and early stopping?
Common mistakes
- Saying every boosting loss trains trees directly on raw residuals.
- Assuming boosting cannot overfit because each tree is small.
When to use it
Use it for tabular prediction where nonlinear patterns and interactions matter. Sequential training and tuning can make it less convenient under tight training budgets.