Back-of-the-Envelope Design for Serving a Vision Model to 100M Daily Users
Problem statement
How would you serve photo tags and embeddings for a very busy consumer app? Most images are small, but bursts of full-resolution album photos must be processed without downscaling. The vision transformer runs on GPUs, and responses must stay fast even when large images strain memory.
Size the compute and memory needs, then explain batching, response time and recovery.
Show hint
What are the three numbers you need before sizing anything: requests per second at peak, FLOPs per request, and bytes held per request?
Relevant roles: ML Engineer, MLOps Engineer and AI Architect.
Related free problems
- GPU Memory Budget for Serving a 70B ModelLLMs & GenAIMedium