Mixture of Experts Load Balancing Loss

MediumML Coding

Problem statement

In a Mixture of Experts (MoE) layer, a gating network routes each token to one (or a few) experts. Left alone, the gate tends to send most tokens to a handful of experts and leave the rest idle, which wastes compute and hurts quality. An auxiliary load balancing loss is added to training to push routing toward an even spread.

Implement moe_load_balancing_loss(gate_logits, num_experts, alpha=0.01):

  • gate_logits: shape (num_tokens, num_experts), raw gate scores per token
  • num_experts: the number of experts $N$
  • alpha: the loss coefficient

Steps, for $T$ tokens:

  1. Softmax each token's logits into routing probabilities $p_{t,i}$ (use a numerically stable softmax).
  2. Hard-route each token to $a_t = \arg\max_i p_{t,i}$.
  3. Dispatch fraction per expert: $f_i = \frac{1}{T}\sum_t \mathbf{1}[a_t = i]$.
  4. Mean gate probability per expert, over all tokens: $P_i = \frac{1}{T}\sum_t p_{t,i}$.
  5. Return $\mathcal{L} = \alpha \cdot N \cdot \sum_i f_i P_i$ as a float rounded to 4 decimals.

Perfectly balanced, confident routing gives a loss of about $\alpha$; imbalanced routing gives more.

Example

Input:

gate_logits = np.array([[10.0, 0.0], [0.0, 10.0]])
moe_load_balancing_loss(gate_logits, num_experts=2, alpha=0.01)

Output:

0.01

Reasoning: the softmax rows are about [0.99995, 0.00005] and [0.00005, 0.99995]. Token 0 goes to expert 0 and token 1 to expert 1, so $f = [0.5, 0.5]$, and the mean probabilities are $P = [0.5, 0.5]$. The loss is $0.01 \cdot 2 \cdot (0.25 + 0.25) = 0.01$, the balanced value.

Show hint

Which quantity tells you how the work was actually split across experts, and which smooth quantity can gradients actually flow through?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer and Research Scientist.

Related free problems