Request Batching for Inference
Problem statement
Implement a request batching function for ML model inference. In production ML systems, batching multiple inference requests together improves throughput by leveraging parallel processing capabilities of hardware accelerators.
Your function should group incoming requests into batches based on two constraints:
- Maximum batch size: A batch should not exceed this number of requests
- Maximum wait time: The time elapsed since the first request in the current batch should not exceed this threshold
When either constraint is violated by an incoming request, the current batch should be finalized and processed, and a new batch should start with the incoming request.
Function Inputs:
requests: A list of dictionaries, each containing:'id': Integer identifier for the request'timestamp': Float representing arrival time in seconds'features': List of float values representing input features
max_batch_size: Integer, maximum number of requests per batchmax_wait_time: Float, maximum seconds to wait before processing a batch
Function Output:
A list of tuples, where each tuple represents a processed batch containing:
- List of request IDs in the batch
- List of feature lists (batched features)
- Processing timestamp (the timestamp of the last request in the batch, rounded to 4 decimals)
Note: Requests should be processed in order of their timestamps, regardless of their order in the input list.
Rules, applied to each request in timestamp order:
- If the current batch is empty, the request starts a new batch.
- Otherwise, if the current batch already holds
max_batch_sizerequests, ortimestamp - first_timestamp_in_batch > max_wait_time(strictly greater), finalize the current batch and start a new one with this request. - Otherwise, add the request to the current batch.
- After the last request, finalize any non-empty batch. Requests with equal timestamps keep their input order. An empty input returns
[].
Example
Input:
requests = [{'id': 1, 'timestamp': 0.0, 'features': [1.0, 2.0]}, {'id': 2, 'timestamp': 0.1, 'features': [3.0, 4.0]}, {'id': 3, 'timestamp': 0.2, 'features': [5.0, 6.0]}, {'id': 4, 'timestamp': 0.3, 'features': [7.0, 8.0]}], max_batch_size = 2, max_wait_time = 10.0
Output:
[([1, 2], [[1.0, 2.0], [3.0, 4.0]], 0.1), ([3, 4], [[5.0, 6.0], [7.0, 8.0]], 0.3)]
Reasoning: Requests arrive at times 0.0, 0.1, 0.2, and 0.3 seconds. With max_batch_size=2, the first two requests (ids 1 and 2) fill the first batch, which is processed at time 0.1. Similarly, requests 3 and 4 form the second batch, processed at time 0.3. The wait time constraint (10.0s) is never triggered since all requests arrive within 0.3 seconds.
Show hint
In what order should requests be considered, given that the input list may not be chronological?
Relevant roles: MLOps Engineer, ML Engineer, AI Engineer and LLM Engineer.