Classify LLM Prefill vs Decode as Compute-Bound or Memory-Bound

MediumML Coding

Problem statement

During LLM inference, there are two distinct phases with very different computational characteristics:

Prefill Phase: The model processes all input tokens simultaneously. Each token requires a forward pass through all model parameters, and since many tokens are processed in a single batch, data loaded from memory is reused across tokens.

Decode Phase: The model generates output tokens one at a time (per sequence). Each generated token still requires loading model weights, but typically processes far fewer tokens per step.

Implement a function that analyzes both phases of LLM inference using the Roofline Model framework. Given the model size, sequence configuration, and hardware specifications, determine:

  • The total FLOPs for each phase (using the standard approximation that each parameter contributes 2 FLOPs per token processed: one multiply and one add)
  • The memory traffic for each phase (total weight bytes that must be loaded from memory, assuming weights are loaded once per phase)
  • The arithmetic intensity (FLOP/byte) for each phase
  • Whether each phase is 'compute-bound' or 'memory-bound' relative to the hardware's ridge point
  • The theoretical achieved performance (FLOP/s) under the roofline model
  • The hardware utilization percentage

When arithmetic intensity equals the ridge point exactly, classify as 'compute-bound'.

Prefill processes sequence_length tokens (one prompt), and one decode step processes batch_size tokens (one new token per sequence in the batch).

Return a dictionary with 'ridge_point' and two sub-dictionaries, 'prefill' and 'decode', each with keys 'total_flops', 'memory_bytes', 'arithmetic_intensity', 'bottleneck', 'achieved_flops' and 'utilization_percent'.

Round all floating-point values in the output to 4 decimal places.

Example

Input:

num_params=2000, sequence_length=4, batch_size=1, bytes_per_param=2, peak_flops=800.0, peak_bandwidth=100.0

Output:

{'ridge_point': 8.0, 'prefill': {'total_flops': 16000, 'memory_bytes': 4000, 'arithmetic_intensity': 4.0, 'bottleneck': 'memory-bound', 'achieved_flops': 400.0, 'utilization_percent': 50.0}, 'decode': {'total_flops': 4000, 'memory_bytes': 4000, 'arithmetic_intensity': 1.0, 'bottleneck': 'memory-bound', 'achieved_flops': 100.0, 'utilization_percent': 12.5}}

Reasoning: Ridge point = 800 / 100 = 8.0 FLOP/byte. Prefill: FLOPs = 2 2000 4 = 16000, memory = 2000 2 = 4000, AI = 16000/4000 = 4.0. Since 4.0 < 8.0, prefill is memory-bound with achieved = 4.0 100 = 400.0 FLOP/s (50% utilization). Decode: FLOPs = 2 2000 1 = 4000, memory = 4000, AI = 1.0. Since 1.0 < 8.0, decode is memory-bound with achieved = 1.0 * 100 = 100.0 FLOP/s (12.5% utilization). With only 4 input tokens, even prefill cannot saturate compute, but decode at batch_size=1 is severely bandwidth-limited.

Show hint

How many tokens share each load of the weights in each phase, and how does that affect FLOPs per byte?

Relevant roles: LLM Engineer, AI Engineer, ML Engineer, Gen AI Engineer and MLOps Engineer.

Related free problems