Designing a Production Inference Pipeline

Reported in interviews at Anthropic Nuro

Problem statement

How would you design an online service that serves several models and frequent updates? Callers name a model and send an entity id and request details, while the service fetches the remaining features. Traffic is high, responses must be fast and the service must stay available through failures. Releases need safe routing, rollout and monitoring.

Explain model serving and releases, diagnose feature differences and choose canary gates.

Show hint

What single indirection lets you change which model version serves which traffic without redeploying callers?

Relevant roles: ML Engineer, MLOps Engineer and AI Engineer.

Related free problems