Testing Whether a New Feature Increases Signups

Reported in interviews at Quora

Problem statement

How would you test whether a signup-page feature increases signups? Only logged-out visitors who reach the page are eligible, and many return across sessions before deciding. The design must keep those repeat visits from distorting assignment or conversion measurement. Plan the experiment and analyze its signup results.

Show hint

Start from what a unit of the experiment is and what exactly you will count as the outcome for that unit.

Relevant roles: Data Scientist and Applied Scientist.

Related free problems