Reading a 2% CTR Lift in a Music Recommendation Test

Reported in interviews at Amazon

Problem statement

How would you decide whether a new music recommendation model succeeded? A user-randomized test shows a small rise in clicks, but more clicked tracks are skipped quickly. Listening time and return rate show no clear improvement, and the goal is music people actually listen to.

Assess uncertainty at the user level, compare sustained listening and decide what to do next.

Show hint

What does "+2%" mean in terms of the two arms' actual rates, and what is the unit that was randomized?

Relevant roles: ML Engineer, Data Scientist and Applied Scientist.

Related free problems