A/B Test Statistical Analysis for Model Comparison
Problem statement
In production ML systems, A/B testing is the standard way to compare model versions. When rolling out a new model (treatment) against an existing model (control), you need a rigorous statistical analysis to make the launch decision.
Given binary outcomes (0 or 1) from both groups, write analyze_ab_test(control_outcomes, treatment_outcomes, confidence_level=0.95, min_detectable_effect=0.02) that returns a dictionary with these keys:
control_rate,treatment_rate: success rates $\hat{p}_c$, $\hat{p}_t$ (rounded to 4 decimals).absolute_lift: $\hat{p}_t - \hat{p}_c$ (4 decimals).relative_lift_pct: $(\hat{p}_t - \hat{p}_c)/\hat{p}_c \cdot 100$ (2 decimals). If $\hat{p}_c = 0$, return the string'inf'when $\hat{p}_t > 0$ and 0.0 otherwise.z_statistic: two-proportion z-test with POOLED variance, $z = \frac{\hat{p}_t - \hat{p}_c}{\sqrt{\hat{p}(1-\hat{p})(1/n_c + 1/n_t)}}$ where $\hat{p}$ is the pooled success rate; use 0 when the pooled rate is 0 or 1 (4 decimals).p_value: two-tailed, $2(1 - \Phi(|z|))$ (4 decimals).confidence_interval: a tuple(lower, upper)for the difference, $(\hat{p}_t - \hat{p}_c) \pm z_{1-\alpha/2}\, SE$, with the UNPOOLED $SE = \sqrt{\hat{p}_c(1-\hat{p}_c)/n_c + \hat{p}_t(1-\hat{p}_t)/n_t}$, where $\alpha = 1 - $confidence_leveland $z_{1-\alpha/2}$ is the exact standard normal quantile (4 decimals each).statistically_significant:p_value < alpha(unrounded).practically_significant:|absolute_lift| >= min_detectable_effect.required_sample_size: per-group size for 80 percent power, $\lceil 2\left(\frac{z_{1-\alpha/2} + 0.84}{\delta}\right)^2 \bar{p}(1-\bar{p}) \rceil$ with $\delta$ =min_detectable_effectand $\bar{p} = (\hat{p}_c + \hat{p}_t)/2$; use 0 if $\delta \le 0$ or $\bar{p}$ is 0 or 1.recommendation:'launch_treatment'if statistically significant AND practically significant AND the lift is positive;- otherwise
'keep_control'if statistically significant AND (the lift is negative OR the effect is not practically significant); - otherwise
'continue_testing'.
If either input list is empty, return an empty dictionary.
Example
Input:
control_outcomes = [1,1,1,0,0,0,1,0,1,0]*50, treatment_outcomes = [1,1,1,1,0,0,1,0,1,0]*50, confidence_level = 0.95, min_detectable_effect = 0.02
Output:
{'control_rate': 0.5, 'treatment_rate': 0.6, 'absolute_lift': 0.1, 'relative_lift_pct': 20.0, 'z_statistic': 3.1782, 'p_value': 0.0015, 'confidence_interval': (0.0386, 0.1614), 'statistically_significant': True, 'practically_significant': True, 'required_sample_size': 9702, 'recommendation': 'launch_treatment'}
Reasoning: Control has 250/500 successes and treatment 300/500. The pooled rate is 0.55, so $z = 0.1/\sqrt{0.55 \cdot 0.45 \cdot (1/500 + 1/500)} \approx 3.178$ and the p-value is about 0.0015 < 0.05. The lift 0.1 exceeds the minimum detectable effect 0.02 and is positive, so the recommendation is to launch the treatment.
Show hint
Under the null hypothesis that the two models perform the same, what single success rate would you estimate from all the data?
Relevant roles: Data Scientist, ML Engineer, Applied Scientist and MLOps Engineer.