Serving a Handed-Over Model Under 200 ms

Reported in interviews at Xometry

Problem statement

How would you put a trained text-and-tabular model behind a prediction API that responds while the user waits? It currently runs in Python on a CPU, pads every input to a fixed maximum and queries the transactional database for features on every request. The latency target is tight, and most inputs are much shorter than the padding length.

Diagnose latency, order optimizations and protect quality on rare categories.

Show hint

Before touching the model, ask what exactly the response time target means and which parts of the request it covers.

Relevant roles: ML Engineer, MLOps Engineer and AI Engineer.

Related free problems