Statistical Power for Experiment Design

MediumML Coding

Problem statement

Statistical power is the probability that a test rejects the null hypothesis when a true effect exists. Implement the power of a two-sample z-test with equal group sizes, the standard back-of-the-envelope calculation for A/B tests.

Write calculate_power(effect_size, sample_size_per_group, alpha=0.05, two_tailed=True) where:

  • effect_size is Cohen's d, $(\mu_1 - \mu_2)/\sigma$
  • sample_size_per_group is $n$, the number of observations in each group
  • alpha is the significance level
  • two_tailed selects a two-sided or a one-sided (upper-tail) test

With the non-centrality parameter $\text{ncp} = d\sqrt{n/2}$ and $\Phi$ the standard normal CDF:

  • two-tailed: $\text{Power} = 1 - \Phi(z_{1-\alpha/2} - \text{ncp}) + \Phi(-z_{1-\alpha/2} - \text{ncp})$
  • one-tailed: $\text{Power} = 1 - \Phi(z_{1-\alpha} - \text{ncp})$

where $z_p = \Phi^{-1}(p)$. Use an accurate normal CDF and quantile (for example statistics.NormalDist from the standard library, or math.erf with a precise inverse); crude approximations can change the 4th decimal. Return the power rounded to 4 decimal places.

Example

Input:

effect_size=0.5, sample_size_per_group=64, alpha=0.05, two_tailed=True

Output:

0.8074

Reasoning: The non-centrality parameter is 0.5 * sqrt(64 / 2) = 2.8284 and the two-sided critical value is 1.96. Power = 1 - Phi(1.96 - 2.8284) + Phi(-1.96 - 2.8284) = 0.8074, about an 81% chance of detecting the effect if it is real.

Show hint

Under the alternative hypothesis, where is the test statistic centered?

Relevant roles: Data Scientist, Applied Scientist and ML Engineer.

Related free problems