Chapter 7: ML system design and experiments

Variance reduction with CUPED

On a real app, a test often waits weeks for enough users, and you cannot always get more traffic. This lesson shows how a number you already have, each user's value from before the test, cuts the users you need. Interviewers ask exactly this question.

20 minInteractive lessonChapter 7: ML system design and experimentsLesson 13 of 13

What you will learn

  • Compute the CUPED-adjusted metric and optimal theta
  • Estimate the variance reduction from the pre/post correlation
  • Explain why CUPED does not bias the treatment effect

Key terms

covariate
a number measured before the test that predicts the metric
CUPED
a method that removes the part of a metric that the covariate predicts
θ (theta)
the slope that says how much of the covariate to remove
ρ (rho)
the correlation between the covariate and the test-week metric

Prerequisites

Take Pitfalls: SRM, peeking, interference and multiple metrics first. It comes right before this lesson in the course.

Where this lesson picks up: You learned to check a finished test for SRM, for early looks, for users who affect each other and for too many metrics. With a clean test, one problem remains: the test can take too long.

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.