Chapter 1: Classical ML
Bagging and random forests
An average of trees shrinks only the part of the wobble that the trees do not share. Interviewers ask why more trees soon help little, what the feature limit per split does, and how to check a forest without a holdout.
What you will learn
- Explain bootstrap sampling and feature subsampling
- Compute the variance of an average of correlated trees and its floor
- Use out-of-bag points as a free validation set
Key terms
- wobble
- how much a tree's prediction changes when the training rows change
- bootstrap sample
- rows drawn from your data with replacement
- out-of-bag row
- a row that a bootstrap sample missed
Prerequisites
Take Decision trees: greedy splits first. It comes right before this lesson in the course.
Where this lesson picks up: You grew decision trees with greedy splits. Deep trees wobble: new training rows give a new prediction. Can many trees together wobble less?
What comes next
An average shrinks variance. Can trees also fix each other's mistakes?
Next lesson: Gradient boosting
More lessons in this part
- Decision trees: greedy splits 19 min
- Gradient boosting 21 min
- Two simple baselines: k-nearest neighbours and Naive Bayes 17 min
- Support vector machines: margins and hinge loss 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.