CUPED: Variance Reduction with Pre-Experiment Data

Problem statement

An A/B test measures a noisy metric, but every user also has a value of the same metric from before the experiment. Because the pre-period value is correlated with the outcome and unaffected by treatment, subtracting the part of the outcome it explains removes noise without changing the expected treatment effect. Implement the CUPED adjustment and the A/B readout that uses it.

cuped_theta(y, x)

  • With y the outcome and x the pre-experiment covariate over the pooled units of both groups, return $\theta = \dfrac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sum_i (x_i - \bar x)^2}$ as a float (0.0 if the denominator is zero).

cuped_adjust(y, x, theta, x_mean)

  • The adjusted outcome $y_i - \theta\,(x_i - \bar x)$, elementwise, where x_mean is the pooled mean of x.

ab_estimate(y_t, y_c)

  • (diff, se): the difference in means, treatment minus control, and the Welch standard error $\sqrt{s_t^2 / n_t + s_c^2 / n_c}$ with sample variances (ddof=1).

cuped_ab(y_t, x_t, y_c, x_c)

  • Pool both groups to compute theta and the pooled mean of x, adjust each group, and return a dict with diff_raw, se_raw, diff_cuped, se_cuped, theta and variance_reduction ($1 - \text{Var}(y_{\text{adj}}) / \text{Var}(y)$ over the pooled units, ddof=1), each rounded to 6 decimals.

sample_size_multiplier(rho)

  • The fraction of the original sample size that reaches the same power after CUPED, $1 - \rho^2$, for a pre-post correlation rho.

Example

Input:

cuped_theta(np.array([2.0, 4.0, 6.0]), np.array([1.0, 2.0, 3.0]))
cuped_adjust(np.array([2.0, 4.0, 6.0]), np.array([1.0, 2.0, 3.0]), 2.0, 2.0)
sample_size_multiplier(0.7)

Output:

2.0
[4.0, 4.0, 4.0]
0.51

Reasoning: the outcome is exactly twice the covariate, so the regression slope is 2 and the adjusted outcomes are all equal: every bit of variation was explained by the pre-period value. A pre-post correlation of 0.7 leaves $1 - 0.49 = 0.51$ of the variance, so about half the sample size reaches the same power.

Show hint

What single number describes how much of the outcome a pre-period covariate can explain linearly?

Relevant roles: Data Scientist, Applied Scientist and ML Engineer.

Related free problems