Choosing K in KNN and the Bias-Variance Tradeoff
Problem statement
- In $k$-nearest-neighbors, how do you choose the value of $k$? Explain how $k$ controls the bias-variance tradeoff.
- Relatedly, in $k$-fold cross-validation, how does the choice of the number of folds $k$ affect the bias and variance of your estimate of test error?
Show hint
What happens at the two extremes of the neighbor count, one neighbor versus every training point?
Relevant roles: Data Scientist, Applied Scientist and ML Engineer.