Serving a Handed-Over Model Under 200 ms
Problem statement
How would you put a trained text-and-tabular model behind a prediction API that responds while the user waits? It currently runs in Python on a CPU, pads every input to a fixed maximum and queries the transactional database for features on every request. The latency target is tight, and most inputs are much shorter than the padding length.
Diagnose latency, order optimizations and protect quality on rare categories.
Show hint
Before touching the model, ask what exactly the response time target means and which parts of the request it covers.
Relevant roles: ML Engineer, MLOps Engineer and AI Engineer.