Sigmoid MoE Router with Bias Correction

HardLLMs & GenAI

Problem statement

Write sigmoid_moe_router(hidden_states, gate_weight, score_bias, top_k) implementing the sigmoid-based Mixture-of-Experts router with a load-balancing bias, as used in recent large MoE language models.

Inputs:

  • hidden_states: array of shape (num_tokens, hidden_dim).
  • gate_weight: array of shape (num_experts, hidden_dim).
  • score_bias: array of shape (num_experts,), used for expert load balancing.
  • top_k: number of experts to select per token.

The function should:

  1. Compute router logits with a matrix multiplication.
  2. Apply a sigmoid (not a softmax) to get routing weights.
  3. Add the score bias to get the scores used for expert selection.
  4. Select the top-k experts per token by those scores, highest first.
  5. Gather the actual sigmoid weights (without the bias) for the selected experts.
  6. Normalize the selected weights so each token's weights sum to 1.

Return a tuple (top_k_weights, top_k_indices), both of shape (num_tokens, top_k). Use only numpy.

Example

Input:

hidden = np.array([[1.0, 0.0], [0.0, 1.0]])
gate = np.array([[1.0, 0.0], [0.0, 1.0], [-1.0, 0.0], [0.0, -1.0]])
bias = np.zeros(4)
w, idx = sigmoid_moe_router(hidden, gate, bias, 2)
print(np.round(w, 4))
print(idx)

Output:

[[0.5938 0.4062]
 [0.5938 0.4062]]
[[0 1]
 [1 0]]

Reasoning: for token [1, 0] the logits are $[1, 0, -1, 0]$ and the sigmoids are about $[0.731, 0.5, 0.269, 0.5]$. The top-2 experts are 0 and 1 with weights $[0.731, 0.5]$, which normalize to $[0.731/1.231, 0.5/1.231] = [0.5938, 0.4062]$.

Show hint

Which activation lets each expert be scored independently instead of making experts compete for one probability budget?

Relevant roles: LLM Engineer, Deep Learning Engineer, ML Engineer and Research Scientist.

Related free problems