Chapter 1: Classical ML
Gradient boosting
You will be able to compute boosting steps by hand and score a tree split the way XGBoost, a popular boosting library, does. Gradient-boosted trees are the usual first choice for tables of data. So interviewers often ask how boosting works and how it differs from a random forest.
What you will learn
- Explain boosting as small gradient steps on residuals
- Compute one boosting step and the XGBoost split gain
- Contrast boosting's bias reduction with bagging's variance reduction
Key terms
- boosting
- adding small trees one after another, each one fixing what is still wrong
- prediction $F$
- the model's current guess for the price
- residual $r$
- the part of the price that the prediction still misses
- learning rate $η$
- the small share of each tree that you add
Prerequisites
Take Bagging and random forests first. It comes right before this lesson in the course.
Where this lesson picks up: You averaged many deep trees, each grown on its own sample, and the average cut variance. But each tree ignored the mistakes of the others, so can trees fix each other's mistakes?
What comes next
Trees, forests, boosting: what other ideas did classic machine learning try?
Next lesson: Two simple baselines: k-nearest neighbours and Naive Bayes
More lessons in this part
- Decision trees: greedy splits 19 min
- Bagging and random forests 25 min
- Two simple baselines: k-nearest neighbours and Naive Bayes 17 min
- Support vector machines: margins and hinge loss 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.