Per-Token Decode Latency from Memory Bandwidth

EasyML Coding

Problem statement

During autoregressive decoding, generating each new token requires streaming all of the model's weights from High Bandwidth Memory (HBM). For large models this weight movement dominates the time per token.

Write a function estimate_decode_latency(num_params, bytes_per_param, bandwidth_bytes_per_s) that returns a list [latency_ms, tokens_per_sec], where latency_ms is the per-token decode latency in milliseconds and tokens_per_sec is the sustained generation throughput in tokens per second.

Inputs:

  • num_params: total number of model parameters (a count, e.g. 7e9 for a 7B model)
  • bytes_per_param: bytes used to store each parameter (e.g. 2 for fp16, 1 for fp8, 4 for fp32)
  • bandwidth_bytes_per_s: HBM memory bandwidth in bytes per second

Return the two values as a list of floats.

Example

Input:

estimate_decode_latency(7e9, 2, 1e12)

Output:

[14.0, 71.4286]

Reasoning: A 7B parameter model stored at 2 bytes per parameter occupies 1.4e10 bytes. Streaming those bytes once at 1e12 bytes/s takes 0.014 s, which is 14.0 ms per token, giving about 71.4286 tokens per second.

Show hint

During single-sequence decoding, what is the GPU mostly waiting on: arithmetic or moving bytes?

Relevant roles: LLM Engineer, AI Engineer, MLOps Engineer and Gen AI Engineer.

Related free problems