Chapter 1: Classical ML

Bagging and random forests

An average of trees shrinks only the part of the wobble that the trees do not share. Interviewers ask why more trees soon help little, what the feature limit per split does, and how to check a forest without a holdout.

25 minInteractive lessonChapter 1: Classical MLLesson 16 of 25

What you will learn

  • Explain bootstrap sampling and feature subsampling
  • Compute the variance of an average of correlated trees and its floor
  • Use out-of-bag points as a free validation set

Key terms

wobble
how much a tree's prediction changes when the training rows change
bootstrap sample
rows drawn from your data with replacement
out-of-bag row
a row that a bootstrap sample missed

Prerequisites

Take Decision trees: greedy splits first. It comes right before this lesson in the course.

Where this lesson picks up: You grew decision trees with greedy splits. Deep trees wobble: new training rows give a new prediction. Can many trees together wobble less?

What comes next

An average shrinks variance. Can trees also fix each other's mistakes?

Next lesson: Gradient boosting

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.