Mixture of Experts Load Balancing Loss
Problem statement
In a Mixture of Experts (MoE) layer, a gating network routes each token to one (or a few) experts. Left alone, the gate tends to send most tokens to a handful of experts and leave the rest idle, which wastes compute and hurts quality. An auxiliary load balancing loss is added to training to push routing toward an even spread.
Implement moe_load_balancing_loss(gate_logits, num_experts, alpha=0.01):
gate_logits: shape(num_tokens, num_experts), raw gate scores per tokennum_experts: the number of experts $N$alpha: the loss coefficient
Steps, for $T$ tokens:
- Softmax each token's logits into routing probabilities $p_{t,i}$ (use a numerically stable softmax).
- Hard-route each token to $a_t = \arg\max_i p_{t,i}$.
- Dispatch fraction per expert: $f_i = \frac{1}{T}\sum_t \mathbf{1}[a_t = i]$.
- Mean gate probability per expert, over all tokens: $P_i = \frac{1}{T}\sum_t p_{t,i}$.
- Return $\mathcal{L} = \alpha \cdot N \cdot \sum_i f_i P_i$ as a float rounded to 4 decimals.
Perfectly balanced, confident routing gives a loss of about $\alpha$; imbalanced routing gives more.
Example
Input:
gate_logits = np.array([[10.0, 0.0], [0.0, 10.0]])
moe_load_balancing_loss(gate_logits, num_experts=2, alpha=0.01)
Output:
0.01
Reasoning: the softmax rows are about [0.99995, 0.00005] and [0.00005, 0.99995]. Token 0 goes to expert 0 and token 1 to expert 1, so $f = [0.5, 0.5]$, and the mean probabilities are $P = [0.5, 0.5]$. The loss is $0.01 \cdot 2 \cdot (0.25 + 0.25) = 0.01$, the balanced value.
Show hint
Which quantity tells you how the work was actually split across experts, and which smooth quantity can gradients actually flow through?
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium