Reading a 2% CTR Lift in a Music Recommendation Test
Problem statement
How would you decide whether a new music recommendation model succeeded? A user-randomized test shows a small rise in clicks, but more clicked tracks are skipped quickly. Listening time and return rate show no clear improvement, and the goal is music people actually listen to.
Assess uncertainty at the user level, compare sustained listening and decide what to do next.
Show hint
What does "+2%" mean in terms of the two arms' actual rates, and what is the unit that was randomized?
Relevant roles: ML Engineer, Data Scientist and Applied Scientist.
Related free problems
- Notebook Click Model: Baseline, Model Comparison and Follow-UpsClassical MLMedium