Routing and Scheduling Layer for Heterogeneous LLM Inference Backends
Problem statement
How would you route requests across a mixed fleet of language-model backends? Tenants include contracted customers, free users and batch jobs, with streaming and non-streaming requests. GPU chat models and small CPU models need fair scheduling, efficient batching and deterministic-result caching, with tenant credits charged by compute used.
Design routing and fair scheduling, then calculate the replica requirement.
Show hint
What unit should priorities, credits and capacity all be measured in when requests differ in cost by orders of magnitude?
Relevant roles: ML Engineer, LLM Engineer, MLOps Engineer and AI Architect.
Related free problems
- LLM Serving: Latency vs Throughput and Batched InferenceLLMs & GenAIMedium