Chapter 7: ML system design and experiments
Variance reduction with CUPED
On a real app, a test often waits weeks for enough users, and you cannot always get more traffic. This lesson shows how a number you already have, each user's value from before the test, cuts the users you need. Interviewers ask exactly this question.
What you will learn
- Compute the CUPED-adjusted metric and optimal theta
- Estimate the variance reduction from the pre/post correlation
- Explain why CUPED does not bias the treatment effect
Key terms
- covariate
- a number measured before the test that predicts the metric
- CUPED
- a method that removes the part of a metric that the covariate predicts
- θ (theta)
- the slope that says how much of the covariate to remove
- ρ (rho)
- the correlation between the covariate and the test-week metric
Prerequisites
Take Pitfalls: SRM, peeking, interference and multiple metrics first. It comes right before this lesson in the course.
Where this lesson picks up: You learned to check a finished test for SRM, for early looks, for users who affect each other and for too many metrics. With a clean test, one problem remains: the test can take too long.
More lessons in this part
- A/B design: randomization unit, metrics and guardrails 20 min
- Sizing: power, minimum detectable effect and duration 21 min
- Many metrics, many tests: Bonferroni, Holm and BH 16 min
- Pitfalls: SRM, peeking, interference and multiple metrics 19 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.