MoE Parameter and Active-Token Memory Estimator
Problem statement
Given a transformer block's embedding dimension and feed-forward (FFN) hidden dimension, estimate the parameter counts and per-token activation memory of a Mixture-of-Experts (MoE) FFN layer and compare it against the equivalent dense FFN.
Implement moe_memory_estimator(emb_dim, hidden_dim, num_experts, top_k, ffn_type='gelu', bytes_per_elem=2, match_dense_budget=False) that returns a dictionary with the following integer fields (in this order):
dense_ffn_params: parameter count of a single dense FFN block usinghidden_dim.per_expert_params: parameter count of one expert (uses the MoE hidden size; see below).router_params: parameters of the linear router that maps a token embedding to expert logits.moe_total_params: total parameters in the MoE FFN layer (all experts + router).active_params_per_token: parameters actually used to process one token (only the chosentop_kexperts plus the router).active_memory_bytes_per_token:active_params_per_token * bytes_per_elem.moe_hidden_dim: the hidden size used inside each expert.
Rules:
- For
ffn_type='gelu', the FFN has two linear projections (no biases): up (emb_dim -> hidden) and down (hidden -> emb_dim), giving2 * emb_dim * hiddenparameters. - For
ffn_type='swiglu', the FFN has three linear projections (gate, up, down), giving3 * emb_dim * hiddenparameters. - The router is a single linear layer of shape
(emb_dim, num_experts)with no bias. - If
match_dense_budget=True, set the expert hidden size tohidden_dim // num_expertsso that the total experts roughly match the dense FFN budget. Otherwise, each expert uses the fullhidden_dim. - Ignore embeddings, attention, and layer norms; only the FFN/MoE block is counted.
Example
Input:
emb_dim=768, hidden_dim=3072, num_experts=8, top_k=2, ffn_type='gelu', bytes_per_elem=2, match_dense_budget=False
Output:
{'dense_ffn_params': 4718592, 'per_expert_params': 4718592, 'router_params': 6144, 'moe_total_params': 37754880, 'active_params_per_token': 9443328, 'active_memory_bytes_per_token': 18886656, 'moe_hidden_dim': 3072}
Reasoning: A GeLU FFN has two linear layers, so dense FFN params = 2 768 3072 = 4,718,592. With 8 experts each at full hidden_dim, total experts = 8 4,718,592 = 37,748,736 plus router (768 8 = 6,144) gives 37,754,880. Each token activates top_k=2 experts plus the router: 2 * 4,718,592 + 6,144 = 9,443,328 params, which at 2 bytes/elem (fp16/bf16) is 18,886,656 bytes.
Show hint
How many weight matrices does each FFN variant have, and what shape is each one?
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, AI Architect and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium