Classify LLM Prefill vs Decode as Compute-Bound or Memory-Bound
Problem statement
During LLM inference, there are two distinct phases with very different computational characteristics:
Prefill Phase: The model processes all input tokens simultaneously. Each token requires a forward pass through all model parameters, and since many tokens are processed in a single batch, data loaded from memory is reused across tokens.
Decode Phase: The model generates output tokens one at a time (per sequence). Each generated token still requires loading model weights, but typically processes far fewer tokens per step.
Implement a function that analyzes both phases of LLM inference using the Roofline Model framework. Given the model size, sequence configuration, and hardware specifications, determine:
- The total FLOPs for each phase (using the standard approximation that each parameter contributes 2 FLOPs per token processed: one multiply and one add)
- The memory traffic for each phase (total weight bytes that must be loaded from memory, assuming weights are loaded once per phase)
- The arithmetic intensity (FLOP/byte) for each phase
- Whether each phase is 'compute-bound' or 'memory-bound' relative to the hardware's ridge point
- The theoretical achieved performance (FLOP/s) under the roofline model
- The hardware utilization percentage
When arithmetic intensity equals the ridge point exactly, classify as 'compute-bound'.
Prefill processes sequence_length tokens (one prompt), and one decode step processes batch_size tokens (one new token per sequence in the batch).
Return a dictionary with 'ridge_point' and two sub-dictionaries, 'prefill' and 'decode', each with keys 'total_flops', 'memory_bytes', 'arithmetic_intensity', 'bottleneck', 'achieved_flops' and 'utilization_percent'.
Round all floating-point values in the output to 4 decimal places.
Example
Input:
num_params=2000, sequence_length=4, batch_size=1, bytes_per_param=2, peak_flops=800.0, peak_bandwidth=100.0
Output:
{'ridge_point': 8.0, 'prefill': {'total_flops': 16000, 'memory_bytes': 4000, 'arithmetic_intensity': 4.0, 'bottleneck': 'memory-bound', 'achieved_flops': 400.0, 'utilization_percent': 50.0}, 'decode': {'total_flops': 4000, 'memory_bytes': 4000, 'arithmetic_intensity': 1.0, 'bottleneck': 'memory-bound', 'achieved_flops': 100.0, 'utilization_percent': 12.5}}
Reasoning: Ridge point = 800 / 100 = 8.0 FLOP/byte. Prefill: FLOPs = 2 2000 4 = 16000, memory = 2000 2 = 4000, AI = 16000/4000 = 4.0. Since 4.0 < 8.0, prefill is memory-bound with achieved = 4.0 100 = 400.0 FLOP/s (50% utilization). Decode: FLOPs = 2 2000 1 = 4000, memory = 4000, AI = 1.0. Since 1.0 < 8.0, decode is memory-bound with achieved = 1.0 * 100 = 100.0 FLOP/s (12.5% utilization). With only 4 input tokens, even prefill cannot saturate compute, but decode at batch_size=1 is severely bandwidth-limited.
Show hint
How many tokens share each load of the weights in each phase, and how does that affect FLOPs per byte?
Relevant roles: LLM Engineer, AI Engineer, ML Engineer, Gen AI Engineer and MLOps Engineer.
Related free problems
- Chunked Prefill Scheduling Alongside DecodeLLMs & GenAIMedium
- Multi-Head AttentionMedium