Classic machine learning

Random forest

Like a panel of judges comparing opinions, a random forest combines many decision trees. Each tree sees a bootstrap sample, and each split considers a random subset of features. Averaging their outputs reduces variance when their errors are not perfectly aligned.

Training dataBootstrapsamplesTree 1P(pass) = 0.8Tree 2P(pass) = 0.6Tree 3P(pass) = 0.4AverageP(pass) = 0.6
For the same input, three trees trained with row and feature randomness give probabilities whose average is 0.6.

How it works

  1. Draw a training sample with replacement for each tree.
  2. At each split, search only a random subset of features.
  3. Grow each tree using its sample and stopping rules.
  4. Average class probabilities for classification or numeric predictions for regression.
  5. Evaluate on held-out data or aggregate out-of-bag predictions from trees that did not sample each row.

$$\hat f(x)=\frac1T\sum_{t=1}^{T}f_t(x)$$

The knobs that matter

n_estimators
More trees stabilize the average but increase memory and prediction work.
max_features
Fewer candidate features can reduce tree correlation but weaken each tree.
min_samples_leaf
Larger leaves smooth each tree and may increase bias.

What interviewers ask

  • Why does averaging trees reduce variance?
  • Why sample features if you already use bootstrap samples?
  • What are out-of-bag predictions, and how would you use them?
  • How does a random forest differ from gradient boosting?

Common mistakes

  • Assuming more trees alone will remove bias from poor features or leaf sizes.
  • Treating impurity-based feature importance as causal or free of bias toward high-cardinality features.

When to use it

Use it as a baseline for tabular data with nonlinear relationships and interactions. Consider other approaches when memory, prediction latency, or regression extrapolation is central.

Implement it (3 problems)

Build it step by step

Learn it

Interview questions about random forest