CUPED: Variance Reduction with Pre-Experiment Data
Problem statement
An A/B test measures a noisy metric, but every user also has a value of the same metric from before the experiment. Because the pre-period value is correlated with the outcome and unaffected by treatment, subtracting the part of the outcome it explains removes noise without changing the expected treatment effect. Implement the CUPED adjustment and the A/B readout that uses it.
cuped_theta(y, x)
- With
ythe outcome andxthe pre-experiment covariate over the pooled units of both groups, return $\theta = \dfrac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sum_i (x_i - \bar x)^2}$ as a float (0.0if the denominator is zero).
cuped_adjust(y, x, theta, x_mean)
- The adjusted outcome $y_i - \theta\,(x_i - \bar x)$, elementwise, where
x_meanis the pooled mean ofx.
ab_estimate(y_t, y_c)
(diff, se): the difference in means, treatment minus control, and the Welch standard error $\sqrt{s_t^2 / n_t + s_c^2 / n_c}$ with sample variances (ddof=1).
cuped_ab(y_t, x_t, y_c, x_c)
- Pool both groups to compute
thetaand the pooled mean ofx, adjust each group, and return a dict withdiff_raw,se_raw,diff_cuped,se_cuped,thetaandvariance_reduction($1 - \text{Var}(y_{\text{adj}}) / \text{Var}(y)$ over the pooled units,ddof=1), each rounded to 6 decimals.
sample_size_multiplier(rho)
- The fraction of the original sample size that reaches the same power after CUPED, $1 - \rho^2$, for a pre-post correlation
rho.
Example
Input:
cuped_theta(np.array([2.0, 4.0, 6.0]), np.array([1.0, 2.0, 3.0]))
cuped_adjust(np.array([2.0, 4.0, 6.0]), np.array([1.0, 2.0, 3.0]), 2.0, 2.0)
sample_size_multiplier(0.7)
Output:
2.0
[4.0, 4.0, 4.0]
0.51
Reasoning: the outcome is exactly twice the covariate, so the regression slope is 2 and the adjusted outcomes are all equal: every bit of variation was explained by the pre-period value. A pre-post correlation of 0.7 leaves $1 - 0.49 = 0.51$ of the variance, so about half the sample size reaches the same power.
Show hint
What single number describes how much of the outcome a pre-period covariate can explain linearly?
Relevant roles: Data Scientist, Applied Scientist and ML Engineer.