MoE Forward Pass with a Shared Expert

MediumML Coding

Problem statement

Implement the forward pass of a Mixture of Experts (MoE) layer that includes a shared expert alongside routed experts. Several recent large MoE language models use this design: one expert always processes every token to maintain shared knowledge, while additional routed experts specialize on different inputs.

Your function takes:

  • X: Input token representations of shape (num_tokens, d_model)
  • W_gate: Gating weight matrix of shape (d_model, num_routed_experts) for computing routing scores
  • W_shared: Shared expert weight matrix of shape (d_model, d_out)
  • W_experts: List of routed expert weight matrices, each of shape (d_model, d_out)
  • top_k: Number of routed experts to activate per token (default 2)

Your function should return a dictionary with the following keys:

  • 'output': Final combined output of shape (num_tokens, d_out)
  • 'shared_output': Output from the shared expert of shape (num_tokens, d_out)
  • 'routed_output': Aggregated output from the routed experts of shape (num_tokens, d_out)
  • 'routing_indices': The selected top-k expert indices per token of shape (num_tokens, top_k), sorted in descending order of gating score
  • 'routing_weights': The renormalized gating weights for the selected experts of shape (num_tokens, top_k)

The gating mechanism should use softmax over all routed experts to produce scores, select the top-k experts per token, renormalize the selected scores to sum to 1, and compute a weighted combination of the selected expert outputs. The shared expert output is added directly to the routed output.

Each expert (shared and routed) performs a simple linear transformation: output = input @ W_expert.

Tie-breaking: When gating scores are equal, select higher expert indices first (equivalent to np.argsort(scores, axis=1)[:, -top_k:][:, ::-1]). Do not use np.argsort(-scores)[:, :top_k], which yields a different order on ties.

Example

Input:

X = np.array([[1.0, 0.0], [0.0, 1.0]]), W_gate = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]]), W_shared = np.eye(2), W_experts = [2*I, [[0,3],[3,0]], [[1,1],[1,1]]], top_k=1

Output:

{'output': [[3.0, 0.0], [3.0, 1.0]], 'routing_indices': [[0], [1]]}

Reasoning: Token 0 ([1,0]) has gate logits [1,0,0]. After softmax, expert 0 has the highest score and is selected. Shared expert output: [1,0] @ I = [1,0]. Routed expert 0 output: [1,0] @ 2I = [2,0]. Final: [1,0]+[2,0] = [3,0]. Token 1 ([0,1]) has gate logits [0,1,0]. Expert 1 is selected. Shared output: [0,1]. Routed expert 1 output: [0,1] @ [[0,3],[3,0]] = [3,0]. Final: [0,1]+[3,0] = [3,1].

Show hint

How do you turn gate logits into a probability over routed experts, and which experts does each token keep?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer and Research Scientist.

Related free problems