Ratio Metrics: Delta-Method Variance and the Unit of Analysis
Problem statement
You changed the ranking model behind search. The primary metric is clicks per query, but the experiment was randomized by user, and each user contributes many queries. Queries from the same user are correlated, so treating every query as an independent observation misstates the variance. The right unit of analysis is the user: sum clicks and queries per user and treat the metric as a ratio of two user-level means. Implement the delta-method variance for that ratio and a two-sample test built on it.
ratio_metric(num, den)
numanddenare per-user totals (for example clicks and queries). Return $\sum_i \text{num}_i / \sum_i \text{den}_i$ rounded to 4 decimals.
delta_method_var(num, den)
- With $n$ users, per-user means $\mu_n$ and $\mu_d$, sample variances $s_n^2$, $s_d^2$ and covariance $s_{nd}$ (all with
ddof=1) and $R = \mu_n / \mu_d$, return $\frac{1}{n\,\mu_d^2}\left(s_n^2 - 2R\,s_{nd} + R^2 s_d^2\right)$ as a float (not rounded).
naive_variance(num, den)
- The query-level Bernoulli variance that ignores users: with $p = \sum \text{num} / \sum \text{den}$ and $N = \sum \text{den}$, return $p(1 - p) / N$ as a float.
ratio_ab_test(num_a, den_a, num_b, den_b)
- Difference of the two ratio metrics (B minus A), its standard error $\sqrt{\text{var}_A + \text{var}_B}$ from the delta method, the z-statistic and the two-sided normal p-value. Return
(diff, se, z, p)rounded to 4 decimals. The normal CDF is $\Phi(z) = \tfrac{1}{2}\left(1 + \text{erf}(z / \sqrt 2)\right)$.
Example
Input:
clicks = [3, 0, 5, 2], queries = [10, 4, 20, 6] (one entry per user)
Output:
ratio_metric = 0.25, delta_method_var = 0.00125, naive_variance = 0.0046875
Reasoning: ten clicks over forty queries is 0.25. The user-level variances are 4.3333 for clicks and 50.6667 for queries with covariance 14, so the delta-method variance is $(4.3333 - 2 \cdot 0.25 \cdot 14 + 0.0625 \cdot 50.6667) / (4 \cdot 100) = 0.00125$, while the query-level formula $0.25 \cdot 0.75 / 40$ ignores the users entirely.
Show hint
Which unit was randomized, and what does that imply about which observations are independent?
Relevant roles: Data Scientist, Applied Scientist and ML Engineer.
Related free problems
- A/B Test Statistical Analysis for Model ComparisonA/B Testing & CausalHard
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium