Sigmoid MoE Router with Bias Correction
Problem statement
Write sigmoid_moe_router(hidden_states, gate_weight, score_bias, top_k) implementing the sigmoid-based Mixture-of-Experts router with a load-balancing bias, as used in recent large MoE language models.
Inputs:
hidden_states: array of shape(num_tokens, hidden_dim).gate_weight: array of shape(num_experts, hidden_dim).score_bias: array of shape(num_experts,), used for expert load balancing.top_k: number of experts to select per token.
The function should:
- Compute router logits with a matrix multiplication.
- Apply a sigmoid (not a softmax) to get routing weights.
- Add the score bias to get the scores used for expert selection.
- Select the top-k experts per token by those scores, highest first.
- Gather the actual sigmoid weights (without the bias) for the selected experts.
- Normalize the selected weights so each token's weights sum to 1.
Return a tuple (top_k_weights, top_k_indices), both of shape (num_tokens, top_k). Use only numpy.
Example
Input:
hidden = np.array([[1.0, 0.0], [0.0, 1.0]])
gate = np.array([[1.0, 0.0], [0.0, 1.0], [-1.0, 0.0], [0.0, -1.0]])
bias = np.zeros(4)
w, idx = sigmoid_moe_router(hidden, gate, bias, 2)
print(np.round(w, 4))
print(idx)
Output:
[[0.5938 0.4062]
[0.5938 0.4062]]
[[0 1]
[1 0]]
Reasoning: for token [1, 0] the logits are $[1, 0, -1, 0]$ and the sigmoids are about $[0.731, 0.5, 0.269, 0.5]$. The top-2 experts are 0 and 1 with weights $[0.731, 0.5]$, which normalize to $[0.731/1.231, 0.5/1.231] = [0.5938, 0.4062]$.
Show hint
Which activation lets each expert be scored independently instead of making experts compete for one probability budget?
Relevant roles: LLM Engineer, Deep Learning Engineer, ML Engineer and Research Scientist.
Related free problems
- MoE Forward Pass with a Shared ExpertML CodingMedium