MoE Parameter and Active-Token Memory Estimator

MediumML Coding

Problem statement

Given a transformer block's embedding dimension and feed-forward (FFN) hidden dimension, estimate the parameter counts and per-token activation memory of a Mixture-of-Experts (MoE) FFN layer and compare it against the equivalent dense FFN.

Implement moe_memory_estimator(emb_dim, hidden_dim, num_experts, top_k, ffn_type='gelu', bytes_per_elem=2, match_dense_budget=False) that returns a dictionary with the following integer fields (in this order):

  • dense_ffn_params: parameter count of a single dense FFN block using hidden_dim.
  • per_expert_params: parameter count of one expert (uses the MoE hidden size; see below).
  • router_params: parameters of the linear router that maps a token embedding to expert logits.
  • moe_total_params: total parameters in the MoE FFN layer (all experts + router).
  • active_params_per_token: parameters actually used to process one token (only the chosen top_k experts plus the router).
  • active_memory_bytes_per_token: active_params_per_token * bytes_per_elem.
  • moe_hidden_dim: the hidden size used inside each expert.

Rules:

  • For ffn_type='gelu', the FFN has two linear projections (no biases): up (emb_dim -> hidden) and down (hidden -> emb_dim), giving 2 * emb_dim * hidden parameters.
  • For ffn_type='swiglu', the FFN has three linear projections (gate, up, down), giving 3 * emb_dim * hidden parameters.
  • The router is a single linear layer of shape (emb_dim, num_experts) with no bias.
  • If match_dense_budget=True, set the expert hidden size to hidden_dim // num_experts so that the total experts roughly match the dense FFN budget. Otherwise, each expert uses the full hidden_dim.
  • Ignore embeddings, attention, and layer norms; only the FFN/MoE block is counted.

Example

Input:

emb_dim=768, hidden_dim=3072, num_experts=8, top_k=2, ffn_type='gelu', bytes_per_elem=2, match_dense_budget=False

Output:

{'dense_ffn_params': 4718592, 'per_expert_params': 4718592, 'router_params': 6144, 'moe_total_params': 37754880, 'active_params_per_token': 9443328, 'active_memory_bytes_per_token': 18886656, 'moe_hidden_dim': 3072}

Reasoning: A GeLU FFN has two linear layers, so dense FFN params = 2 768 3072 = 4,718,592. With 8 experts each at full hidden_dim, total experts = 8 4,718,592 = 37,748,736 plus router (768 8 = 6,144) gives 37,754,880. Each token activates top_k=2 experts plus the router: 2 * 4,718,592 + 6,144 = 9,443,328 params, which at 2 bytes/elem (fp16/bf16) is 18,886,656 bytes.

Show hint

How many weight matrices does each FFN variant have, and what shape is each one?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, AI Architect and Research Scientist.

Related free problems