Statistical Power for Experiment Design
Problem statement
Statistical power is the probability that a test rejects the null hypothesis when a true effect exists. Implement the power of a two-sample z-test with equal group sizes, the standard back-of-the-envelope calculation for A/B tests.
Write calculate_power(effect_size, sample_size_per_group, alpha=0.05, two_tailed=True) where:
effect_sizeis Cohen's d, $(\mu_1 - \mu_2)/\sigma$sample_size_per_groupis $n$, the number of observations in each groupalphais the significance leveltwo_tailedselects a two-sided or a one-sided (upper-tail) test
With the non-centrality parameter $\text{ncp} = d\sqrt{n/2}$ and $\Phi$ the standard normal CDF:
- two-tailed: $\text{Power} = 1 - \Phi(z_{1-\alpha/2} - \text{ncp}) + \Phi(-z_{1-\alpha/2} - \text{ncp})$
- one-tailed: $\text{Power} = 1 - \Phi(z_{1-\alpha} - \text{ncp})$
where $z_p = \Phi^{-1}(p)$. Use an accurate normal CDF and quantile (for example statistics.NormalDist from the standard library, or math.erf with a precise inverse); crude approximations can change the 4th decimal. Return the power rounded to 4 decimal places.
Example
Input:
effect_size=0.5, sample_size_per_group=64, alpha=0.05, two_tailed=True
Output:
0.8074
Reasoning: The non-centrality parameter is 0.5 * sqrt(64 / 2) = 2.8284 and the two-sided critical value is 1.96. Power = 1 - Phi(1.96 - 2.8284) + Phi(-1.96 - 2.8284) = 0.8074, about an 81% chance of detecting the effect if it is real.
Show hint
Under the alternative hypothesis, where is the test statistic centered?
Relevant roles: Data Scientist, Applied Scientist and ML Engineer.
Related free problems
- Factors That Do Not Affect Test PowerStatistics & InferenceEasy
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium