Chapter 1: Classical ML

Gradient boosting

You will be able to compute boosting steps by hand and score a tree split the way XGBoost, a popular boosting library, does. Gradient-boosted trees are the usual first choice for tables of data. So interviewers often ask how boosting works and how it differs from a random forest.

21 minInteractive lessonChapter 1: Classical MLLesson 17 of 25

What you will learn

  • Explain boosting as small gradient steps on residuals
  • Compute one boosting step and the XGBoost split gain
  • Contrast boosting's bias reduction with bagging's variance reduction

Key terms

boosting
adding small trees one after another, each one fixing what is still wrong
prediction $F$
the model's current guess for the price
residual $r$
the part of the price that the prediction still misses
learning rate $η$
the small share of each tree that you add

Prerequisites

Take Bagging and random forests first. It comes right before this lesson in the course.

Where this lesson picks up: You averaged many deep trees, each grown on its own sample, and the average cut variance. But each tree ignored the mistakes of the others, so can trees fix each other's mistakes?

What comes next

Trees, forests, boosting: what other ideas did classic machine learning try?

Next lesson: Two simple baselines: k-nearest neighbours and Naive Bayes

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.