Per-Token Decode Latency from Memory Bandwidth
Problem statement
During autoregressive decoding, generating each new token requires streaming all of the model's weights from High Bandwidth Memory (HBM). For large models this weight movement dominates the time per token.
Write a function estimate_decode_latency(num_params, bytes_per_param, bandwidth_bytes_per_s) that returns a list [latency_ms, tokens_per_sec], where latency_ms is the per-token decode latency in milliseconds and tokens_per_sec is the sustained generation throughput in tokens per second.
Inputs:
- num_params: total number of model parameters (a count, e.g. 7e9 for a 7B model)
- bytes_per_param: bytes used to store each parameter (e.g. 2 for fp16, 1 for fp8, 4 for fp32)
- bandwidth_bytes_per_s: HBM memory bandwidth in bytes per second
Return the two values as a list of floats.
Example
Input:
estimate_decode_latency(7e9, 2, 1e12)
Output:
[14.0, 71.4286]
Reasoning: A 7B parameter model stored at 2 bytes per parameter occupies 1.4e10 bytes. Streaming those bytes once at 1e12 bytes/s takes 0.014 s, which is 14.0 ms per token, giving about 71.4286 tokens per second.
Show hint
During single-sequence decoding, what is the GPU mostly waiting on: arithmetic or moving bytes?
Relevant roles: LLM Engineer, AI Engineer, MLOps Engineer and Gen AI Engineer.
Related free problems
- LLM Serving: Latency vs Throughput and Batched InferenceLLMs & GenAIMedium
- Multi-Head AttentionMedium