Chapter 1: Classical ML
Explaining a model: permutation importance and SHAP
You will measure how much a model relies on each column, and explain one prediction column by column. You will also spot the cases where both answers mislead, which interviewers like to ask about.
What you will learn
- Compute permutation importance on held-out data
- Read a SHAP waterfall: base value plus contributions equals the prediction
- Explain how correlated features distort both methods, and why impurity importances are biased
Key terms
- ลท (y-hat)
- the model's prediction; y is the real value
- MSE
- mean squared error: the average of $(y - \hat y)^2$ over the rows
- held-out rows
- rows the model never trained on
Prerequisites
Take Prediction intervals and conformal prediction first. It comes right before this lesson in the course.
Where this lesson picks up: You gave predictions an interval. The next question is why the model makes its predictions.
What comes next
Everything so far had labels. What if nobody has labelled the data?
Next lesson: Clustering: k-means, restarts and k-means++
More lessons in this part
- ROC, PR curves and AUC 15 min
- Calibration and class imbalance 20 min
- Prediction intervals and conformal prediction 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.