GPT FeedForward Block (Linear-GELU-Linear)
Problem statement
Implement the position-wise feedforward block used inside a GPT transformer layer. Given an input tensor of shape (batch_size, num_tokens, emb_dim), the block applies:
- A linear projection from
emb_dimto4 * emb_dim(weightW1of shape(emb_dim, 4*emb_dim), biasb1of shape(4*emb_dim,)). - A GELU activation, using the tanh approximation from GPT-2.
- A linear projection from
4 * emb_dimback toemb_dim(weightW2of shape(4*emb_dim, emb_dim), biasb2of shape(emb_dim,)).
The tanh-approximate GELU is
$$\mathrm{GELU}(x) = 0.5\, x \left(1 + \tanh\!\left(\sqrt{2/\pi}\,\big(x + 0.044715\, x^3\big)\right)\right).$$
Implement gpt_feedforward(x, W1, b1, W2, b2) and return a numpy array with the same shape as the input. Rounding is only used in test comparisons; do not round inside your implementation.
Example
Input:
x = [[[1.0, -1.0]]] # shape (1, 1, 2)
W1 = [[1.0, 0.0, -1.0, 0.5],
[0.0, 1.0, 1.0, -0.5]]
b1 = [0, 0, 0, 0]
W2 = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0], [-1.0, 0.5]]
b2 = [0, 0]
Output:
[[[-0.0454, 0.2164]]]
Reasoning: the hidden pre-activation is h = x @ W1 + b1 = [1, -1, -2, 1]. GELU gives about [0.8412, -0.1588, -0.0454, 0.8412]. Projecting back with W2 and adding b2 gives about [-0.0454, 0.2164], with the same (1, 1, 2) shape as x.
Show hint
Which axis of the input do the weights act on, and do you need to treat tokens or batch elements separately?
Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium