Tensor Parallelism All-Reduce Communication Cost

EasyML Coding

Problem statement

When training or serving large neural networks with tensor parallelism, individual layers (such as linear projections in transformers) are split across multiple GPUs. After each GPU computes its partial result, an all-reduce collective communication operation is required to synchronize and combine the partial outputs across all participating GPUs.

Understanding the communication cost of these all-reduce operations is critical for performance analysis, since communication overhead can become the bottleneck in multi-GPU systems.

Implement a function tensor_parallel_allreduce_cost that computes the communication cost analysis for tensor-parallel training/inference of a transformer model.

Inputs:

  • hidden_size: The hidden dimension of the model (int)
  • sequence_length: The sequence length (int)
  • batch_size: Micro-batch size per GPU (int)
  • num_gpus: Number of GPUs in the tensor parallel group (int)
  • num_layers: Number of transformer layers (int)
  • bytes_per_element: Bytes per parameter/activation element, e.g. 2 for FP16 (int, default 2)
  • bandwidth_gb_per_sec: Inter-GPU bandwidth in GB/s (float, default 300.0)
  • allreduces_per_layer: Number of all-reduce operations per transformer layer in a single forward pass (int, default 2)

Output: A dictionary with:

  • 'message_size_bytes': Size in bytes of the activation tensor being all-reduced (int)
  • 'comm_volume_per_allreduce_bytes': Communication volume per GPU for a single all-reduce using the ring algorithm (float, rounded to 4 decimals)
  • 'total_comm_volume_bytes': Total communication volume across all layers (float, rounded to 4 decimals)
  • 'total_comm_time_ms': Total communication time in milliseconds (float, rounded to 4 decimals)

The activation tensor that gets all-reduced has shape (batch_size, sequence_length, hidden_size). The ring all-reduce algorithm is a standard approach where each GPU sends and receives data in a ring topology, and you should compute the per-GPU communication volume accordingly.

Use 1 GB/s = $10^9$ bytes per second. Note: when there is only 1 GPU, no communication is needed.

Example

Input:

hidden_size=512, sequence_length=128, batch_size=1, num_gpus=2, num_layers=1, bytes_per_element=2, bandwidth_gb_per_sec=100.0, allreduces_per_layer=2

Output:

{'message_size_bytes': 131072, 'comm_volume_per_allreduce_bytes': 131072.0, 'total_comm_volume_bytes': 262144.0, 'total_comm_time_ms': 0.0026}

Reasoning: The activation tensor has shape (1, 128, 512) with 65536 elements. At 2 bytes per element, the message size is 131072 bytes. With 2 GPUs using ring all-reduce, the per-GPU communication volume is 2 (2-1)/2 131072 = 131072 bytes. With 2 all-reduces per layer and 1 layer, the total communication volume is 262144 bytes. At 100 GB/s bandwidth, the communication time is 262144 / (100 10^9) 1000 = 0.0026 ms.

Show hint

Which tensor do the GPUs need to combine after a row-split matmul, and how many bytes does it occupy?

Relevant roles: MLOps Engineer, ML Engineer, LLM Engineer, AI Architect and Deep Learning Engineer.

Related free problems