Designing a MapReduce Pipeline for Daily User Features

Reported in interviews at Anthropic

Problem statement

How would you build daily user features with MapReduce? A very large event stream feeds historical training data and an online store that needs the latest day. The nightly job has a tight completion window and must stay correct through retries and failures.

Design storage, safe output commits and an efficient, measurable pipeline.

Show hint

Follow the bytes: what each stage reads, what it sends over the network, what it writes, and who reads the output afterwards and how.

Relevant roles: ML Engineer and MLOps Engineer.

Related free problems