Bayesian A/B Test for Conversion Rates

Reported in interviews at Barclays

Problem statement

You are running an A/B test. Variant A has $h_A$ hits (conversions) out of $n_A$ trials, and variant B has $h_B$ hits out of $n_B$ trials. Assume independent $\text{Beta}(1,1)$ (i.e., uniform) priors on the true conversion rates $p_A$ and $p_B$.

(a) Derive the posterior probability $P(p_A > p_B \mid \text{data})$ and express it in terms of Beta functions (or as a one-dimensional integral).

(b) Propose a decision rule for declaring a winner based on this posterior probability.

(c) Construct a Bayesian credible interval for the difference $p_A - p_B$.

Show hint

What posterior does a uniform prior produce for a rate after observing Binomial data, and are the two variants' posteriors linked?

Relevant roles: Data Scientist, Applied Scientist and ML Engineer.

Related free problems