MoE Forward Pass with a Shared Expert
Problem statement
Implement the forward pass of a Mixture of Experts (MoE) layer that includes a shared expert alongside routed experts. Several recent large MoE language models use this design: one expert always processes every token to maintain shared knowledge, while additional routed experts specialize on different inputs.
Your function takes:
X: Input token representations of shape(num_tokens, d_model)W_gate: Gating weight matrix of shape(d_model, num_routed_experts)for computing routing scoresW_shared: Shared expert weight matrix of shape(d_model, d_out)W_experts: List of routed expert weight matrices, each of shape(d_model, d_out)top_k: Number of routed experts to activate per token (default 2)
Your function should return a dictionary with the following keys:
'output': Final combined output of shape(num_tokens, d_out)'shared_output': Output from the shared expert of shape(num_tokens, d_out)'routed_output': Aggregated output from the routed experts of shape(num_tokens, d_out)'routing_indices': The selected top-k expert indices per token of shape(num_tokens, top_k), sorted in descending order of gating score'routing_weights': The renormalized gating weights for the selected experts of shape(num_tokens, top_k)
The gating mechanism should use softmax over all routed experts to produce scores, select the top-k experts per token, renormalize the selected scores to sum to 1, and compute a weighted combination of the selected expert outputs. The shared expert output is added directly to the routed output.
Each expert (shared and routed) performs a simple linear transformation: output = input @ W_expert.
Tie-breaking: When gating scores are equal, select higher expert indices first (equivalent to np.argsort(scores, axis=1)[:, -top_k:][:, ::-1]). Do not use np.argsort(-scores)[:, :top_k], which yields a different order on ties.
Example
Input:
X = np.array([[1.0, 0.0], [0.0, 1.0]]), W_gate = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]]), W_shared = np.eye(2), W_experts = [2*I, [[0,3],[3,0]], [[1,1],[1,1]]], top_k=1
Output:
{'output': [[3.0, 0.0], [3.0, 1.0]], 'routing_indices': [[0], [1]]}
Reasoning: Token 0 ([1,0]) has gate logits [1,0,0]. After softmax, expert 0 has the highest score and is selected. Shared expert output: [1,0] @ I = [1,0]. Routed expert 0 output: [1,0] @ 2I = [2,0]. Final: [1,0]+[2,0] = [3,0]. Token 1 ([0,1]) has gate logits [0,1,0]. Expert 1 is selected. Shared output: [0,1]. Routed expert 1 output: [0,1] @ [[0,3],[3,0]] = [3,0]. Final: [0,1]+[3,0] = [3,1].
Show hint
How do you turn gate logits into a probability over routed experts, and which experts does each token keep?
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium