Classic machine learning
Random forest
Like a panel of judges comparing opinions, a random forest combines many decision trees. Each tree sees a bootstrap sample, and each split considers a random subset of features. Averaging their outputs reduces variance when their errors are not perfectly aligned.
How it works
- Draw a training sample with replacement for each tree.
- At each split, search only a random subset of features.
- Grow each tree using its sample and stopping rules.
- Average class probabilities for classification or numeric predictions for regression.
- Evaluate on held-out data or aggregate out-of-bag predictions from trees that did not sample each row.
$$\hat f(x)=\frac1T\sum_{t=1}^{T}f_t(x)$$
The knobs that matter
- n_estimators
- More trees stabilize the average but increase memory and prediction work.
- max_features
- Fewer candidate features can reduce tree correlation but weaken each tree.
- min_samples_leaf
- Larger leaves smooth each tree and may increase bias.
What interviewers ask
- Why does averaging trees reduce variance?
- Why sample features if you already use bootstrap samples?
- What are out-of-bag predictions, and how would you use them?
- How does a random forest differ from gradient boosting?
Common mistakes
- Assuming more trees alone will remove bias from poor features or leaf sizes.
- Treating impurity-based feature importance as causal or free of bias toward high-cardinality features.
When to use it
Use it as a baseline for tabular data with nonlinear relationships and interactions. Consider other approaches when memory, prediction latency, or regression extrapolation is central.
Implement it (3 problems)
Build it step by step
- Random Forest from Scratch15 steps, your code runs in the browser