Back-of-the-Envelope Design for Serving a Vision Model to 100M Daily Users

Reported in interviews at Waymo

Problem statement

How would you serve photo tags and embeddings for a very busy consumer app? Most images are small, but bursts of full-resolution album photos must be processed without downscaling. The vision transformer runs on GPUs, and responses must stay fast even when large images strain memory.

Size the compute and memory needs, then explain batching, response time and recovery.

Show hint

What are the three numbers you need before sizing anything: requests per second at peak, FLOPs per request, and bytes held per request?

Relevant roles: ML Engineer, MLOps Engineer and AI Architect.

Related free problems