Designing a Production Inference Pipeline
Problem statement
How would you design an online service that serves several models and frequent updates? Callers name a model and send an entity id and request details, while the service fetches the remaining features. Traffic is high, responses must be fast and the service must stay available through failures. Releases need safe routing, rollout and monitoring.
Explain model serving and releases, diagnose feature differences and choose canary gates.
Show hint
What single indirection lets you change which model version serves which traffic without redeploying callers?
Relevant roles: ML Engineer, MLOps Engineer and AI Engineer.