Neural Network Layers From Scratch: ReLU, Stable Softmax, LayerNorm, BatchNorm, RMSNorm

MediumML Coding

Reported in interviews at Startups Com

Problem statement

In a hand-coding round you are asked to build the core layers of a network yourself, using only basic tensor operations (elementwise math, max, sum, mean, exp, sqrt, reshape and broadcasting). Calling a library implementation of the same layer is not allowed. The judge here runs numpy; each line you write maps directly onto the matching PyTorch tensor op (x.max(dim, keepdim=True), x.mean(dim), and so on).

Implement one dispatcher:

def layer_forward(op: str, x: list, params: dict) -> list:
    ...

x is a nested list of floats; convert it with np.asarray(x, dtype=np.float64). The ops are:

  1. "relu" (params = {}): $y = \max(x, 0)$ elementwise.
  2. "softmax" (params = {"dim": d}, d may be negative): softmax along axis d. Inputs can be as large as $\pm 1000$ and the result must stay finite.
  3. "layernorm" (params = {"gamma", "beta", "eps"}): normalize each vector along the LAST axis with its own mean $\mu$ and biased variance $\sigma^2$ (divide by $D$), then $y = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \cdot \gamma + \beta$, with $\gamma, \beta$ of length $D$.
  4. "rmsnorm" (params = {"gamma", "eps"}): along the last axis, $y = \frac{x}{\sqrt{\mathrm{mean}(x^2) + \epsilon}} \cdot \gamma$. No mean subtraction, no bias.
  5. "batchnorm" (params = {"gamma", "beta", "eps", "momentum", "running_mean", "running_var", "training"}): x has shape (N, C) or (N, C, ...) (for example (N, C, H, W)); statistics are per channel (axis 1), taken over every other axis.
    • Training: normalize with the batch mean and the BIASED batch variance, then update the buffers with the convention running_mean = (1 - momentum) * running_mean + momentum * batch_mean and running_var = (1 - momentum) * running_var + momentum * unbiased_batch_var, where the unbiased variance divides by $n - 1$ and $n$ is the number of values per channel.
    • Eval: normalize with the given running statistics and return them unchanged.

Return y as a nested list for every op except "batchnorm", which returns [y, running_mean, running_var].

After coding, explain how you would convince yourself that each layer is correct (reference comparisons, gradient checks, and the edge cases you would test).

Constraints

  • Up to 4 dimensions, at most a few hundred elements per call.
  • In training mode every channel has $n \ge 2$ values.
  • Outputs are compared with a relative tolerance of $10^{-6}$.

Examples

  • layer_forward("softmax", [[1000.0, 1001.0, 1002.0]], {"dim": -1}) -> [[0.0900, 0.2447, 0.6652]] (rounded). A direct exp(1000) overflows to infinity.
  • layer_forward("rmsnorm", [[3.0, 4.0]], {"gamma": [1.0, 2.0], "eps": 0.0}) -> [[0.8485, 2.2627]] (rounded): the root mean square is $\sqrt{12.5} \approx 3.536$.
  • layer_forward("batchnorm", [[1.0, 2.0], [3.0, 6.0]], {"gamma": [1.0, 1.0], "beta": [0.0, 0.0], "eps": 1e-5, "momentum": 0.1, "running_mean": [0.0, 0.0], "running_var": [1.0, 1.0], "training": True}) -> [[[-1.0, -1.0], [1.0, 1.0]], [0.2, 0.4], [1.1, 1.7]] (the outputs are within $10^{-5}$ of $\pm 1$ because of $\epsilon$). The batch means are 2 and 4; the stored variances use the unbiased estimates 2 and 8.
Show hint

For each layer, which axis is reduced, and does the layer behave differently at training and inference time?

Relevant roles: ML Engineer, Deep Learning Engineer and Research Scientist.

Related free problems