GPT FeedForward Block (Linear-GELU-Linear)

EasyML Coding

Problem statement

Implement the position-wise feedforward block used inside a GPT transformer layer. Given an input tensor of shape (batch_size, num_tokens, emb_dim), the block applies:

  1. A linear projection from emb_dim to 4 * emb_dim (weight W1 of shape (emb_dim, 4*emb_dim), bias b1 of shape (4*emb_dim,)).
  2. A GELU activation, using the tanh approximation from GPT-2.
  3. A linear projection from 4 * emb_dim back to emb_dim (weight W2 of shape (4*emb_dim, emb_dim), bias b2 of shape (emb_dim,)).

The tanh-approximate GELU is

$$\mathrm{GELU}(x) = 0.5\, x \left(1 + \tanh\!\left(\sqrt{2/\pi}\,\big(x + 0.044715\, x^3\big)\right)\right).$$

Implement gpt_feedforward(x, W1, b1, W2, b2) and return a numpy array with the same shape as the input. Rounding is only used in test comparisons; do not round inside your implementation.

Example

Input:

x  = [[[1.0, -1.0]]]                       # shape (1, 1, 2)
W1 = [[1.0, 0.0, -1.0,  0.5],
      [0.0, 1.0,  1.0, -0.5]]
b1 = [0, 0, 0, 0]
W2 = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0], [-1.0, 0.5]]
b2 = [0, 0]

Output:

[[[-0.0454, 0.2164]]]

Reasoning: the hidden pre-activation is h = x @ W1 + b1 = [1, -1, -2, 1]. GELU gives about [0.8412, -0.1588, -0.0454, 0.8412]. Projecting back with W2 and adding b2 gives about [-0.0454, 0.2164], with the same (1, 1, 2) shape as x.

Show hint

Which axis of the input do the weights act on, and do you need to treat tokens or batch elements separately?

Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and AI Engineer.

Related free problems