Decompose a Metric Change into Mix Shift and Within-Segment Change
Problem statement
Engagement fell five percent month over month. Before anyone blames the product, split the change into two parts: how much came from the metric moving within each user segment, and how much came from the segments changing size. Implement the standard two-term decomposition of a weighted average.
Inputs are aligned arrays over segments: user counts n0, n1 and per-segment rates r0, r1 for the before and after periods.
overall_rate(n, r)
- The count-weighted average $\sum_i w_i r_i$ with $w_i = n_i / \sum_j n_j$.
decompose_change(n0, r0, n1, r1)
- With weights $w^0$ and $w^1$, return a dict with
total($R^1 - R^0$),mix_effect($\sum_i (w^1_i - w^0_i)\, r^0_i$: the change in segment shares valued at the old rates),rate_effect($\sum_i w^1_i\,(r^1_i - r^0_i)$: the change in rates valued at the new shares), and two listsmix_contributionsandrate_contributionswith the per-segment terms. The two effects sum to the total exactly. Round every number to 6 decimals.
top_contributors(contributions, k)
- Indices of the
ksegments with the largest absolute contribution, largest first (lower index first on ties).
is_simpson(n0, r0, n1, r1)
Truewhen every segment's rate did not fall but the overall rate fell, or every segment's rate did not rise but the overall rate rose: a change explained entirely by mix.
report(names, n0, r0, n1, r1)
- A list of lines
f"{name}: share {w0:.1%} -> {w1:.1%}, rate {r0:.3f} -> {r1:.3f}, mix {mix:+.4f}, rate effect {rate:+.4f}", one per segment in input order, thenf"total {total:+.4f} = mix {mix_effect:+.4f} + rate {rate_effect:+.4f}".
Example
Input:
decompose_change(n0=[800, 200], r0=[0.5, 0.2], n1=[500, 500], r1=[0.5, 0.2])
Output:
{'total': -0.09, 'mix_effect': -0.09, 'rate_effect': 0.0, 'mix_contributions': [-0.15, 0.06], 'rate_contributions': [0.0, 0.0]}
Reasoning: no segment's rate changed, but the low-rate segment grew from 20 percent to 50 percent of users. The overall rate falls from 0.44 to 0.35, and the whole nine-point drop is mix.
Show hint
What are the two ways a count-weighted average can change if you hold the formula fixed?
Relevant roles: Data Scientist, Applied Scientist and ML Engineer.
Related free problems
- Detecting and Handling Data Shift in Deployed ModelsML System DesignMedium