Neural Network Layers From Scratch: ReLU, Stable Softmax, LayerNorm, BatchNorm, RMSNorm
Problem statement
In a hand-coding round you are asked to build the core layers of a network yourself, using only basic tensor operations (elementwise math, max, sum, mean, exp, sqrt, reshape and broadcasting). Calling a library implementation of the same layer is not allowed. The judge here runs numpy; each line you write maps directly onto the matching PyTorch tensor op (x.max(dim, keepdim=True), x.mean(dim), and so on).
Implement one dispatcher:
def layer_forward(op: str, x: list, params: dict) -> list:
...
x is a nested list of floats; convert it with np.asarray(x, dtype=np.float64). The ops are:
"relu"(params = {}): $y = \max(x, 0)$ elementwise."softmax"(params = {"dim": d},dmay be negative): softmax along axisd. Inputs can be as large as $\pm 1000$ and the result must stay finite."layernorm"(params = {"gamma", "beta", "eps"}): normalize each vector along the LAST axis with its own mean $\mu$ and biased variance $\sigma^2$ (divide by $D$), then $y = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \cdot \gamma + \beta$, with $\gamma, \beta$ of length $D$."rmsnorm"(params = {"gamma", "eps"}): along the last axis, $y = \frac{x}{\sqrt{\mathrm{mean}(x^2) + \epsilon}} \cdot \gamma$. No mean subtraction, no bias."batchnorm"(params = {"gamma", "beta", "eps", "momentum", "running_mean", "running_var", "training"}):xhas shape(N, C)or(N, C, ...)(for example(N, C, H, W)); statistics are per channel (axis 1), taken over every other axis.- Training: normalize with the batch mean and the BIASED batch variance, then update the buffers with the convention
running_mean = (1 - momentum) * running_mean + momentum * batch_meanandrunning_var = (1 - momentum) * running_var + momentum * unbiased_batch_var, where the unbiased variance divides by $n - 1$ and $n$ is the number of values per channel. - Eval: normalize with the given running statistics and return them unchanged.
- Training: normalize with the batch mean and the BIASED batch variance, then update the buffers with the convention
Return y as a nested list for every op except "batchnorm", which returns [y, running_mean, running_var].
After coding, explain how you would convince yourself that each layer is correct (reference comparisons, gradient checks, and the edge cases you would test).
Constraints
- Up to 4 dimensions, at most a few hundred elements per call.
- In training mode every channel has $n \ge 2$ values.
- Outputs are compared with a relative tolerance of $10^{-6}$.
Examples
layer_forward("softmax", [[1000.0, 1001.0, 1002.0]], {"dim": -1})->[[0.0900, 0.2447, 0.6652]](rounded). A directexp(1000)overflows to infinity.layer_forward("rmsnorm", [[3.0, 4.0]], {"gamma": [1.0, 2.0], "eps": 0.0})->[[0.8485, 2.2627]](rounded): the root mean square is $\sqrt{12.5} \approx 3.536$.layer_forward("batchnorm", [[1.0, 2.0], [3.0, 6.0]], {"gamma": [1.0, 1.0], "beta": [0.0, 0.0], "eps": 1e-5, "momentum": 0.1, "running_mean": [0.0, 0.0], "running_var": [1.0, 1.0], "training": True})->[[[-1.0, -1.0], [1.0, 1.0]], [0.2, 0.4], [1.1, 1.7]](the outputs are within $10^{-5}$ of $\pm 1$ because of $\epsilon$). The batch means are 2 and 4; the stored variances use the unbiased estimates 2 and 8.
Show hint
For each layer, which axis is reduced, and does the layer behave differently at training and inference time?
Relevant roles: ML Engineer, Deep Learning Engineer and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium