Bayesian A/B Test for Conversion Rates
Problem statement
You are running an A/B test. Variant A has $h_A$ hits (conversions) out of $n_A$ trials, and variant B has $h_B$ hits out of $n_B$ trials. Assume independent $\text{Beta}(1,1)$ (i.e., uniform) priors on the true conversion rates $p_A$ and $p_B$.
(a) Derive the posterior probability $P(p_A > p_B \mid \text{data})$ and express it in terms of Beta functions (or as a one-dimensional integral).
(b) Propose a decision rule for declaring a winner based on this posterior probability.
(c) Construct a Bayesian credible interval for the difference $p_A - p_B$.
Show hint
What posterior does a uniform prior produce for a rate after observing Binomial data, and are the two variants' posteriors linked?
Relevant roles: Data Scientist, Applied Scientist and ML Engineer.