Batch Normalization for BCHW Input
Problem statement
Implement Batch Normalization for a 4D NumPy array of feature maps in BCHW format (batch, channels, height, width). gamma and beta have shape (1, C, 1, 1), and so do the running statistics.
Training mode (training=True):
- For each channel, compute the mean and the variance of the current batch over the batch and spatial axes (B, H, W). Use the population variance (divide by the number of elements, as
np.vardoes). - Normalize with these batch statistics: $\hat{x} = (x - \mu) / \sqrt{\sigma^2 + \epsilon}$.
- Update the running statistics with momentum:
running_stat = (1 - momentum) * running_stat + momentum * batch_stat.
Inference mode (training=False):
- Normalize with the provided running mean and variance (do not compute statistics from the batch) and leave them unchanged.
In both modes, then apply the scale and shift: $y = \gamma \hat{x} + \beta$. If running_mean or running_var is None, initialize them to zeros and ones respectively.
Return a tuple (output, running_mean, running_var).
Example
Input:
X = np.array([[[[1.0, 3.0]]], [[[5.0, 7.0]]]]) # shape (2, 1, 1, 2)
gamma = np.ones((1, 1, 1, 1)); beta = np.zeros((1, 1, 1, 1))
out, rm, rv = batch_normalization(X, gamma, beta, training=True)
print(np.round(out, 4).tolist(), rm.tolist(), rv.tolist())
Output:
[[[[-1.3416, -0.4472]]], [[[0.4472, 1.3416]]]] [[[[0.4]]]] [[[[1.4]]]]
Reasoning: The single channel has values $1, 3, 5, 7$ with mean 4 and variance 5, so the normalized values are $(x - 4)/\sqrt{5}$. The running mean becomes $0.9 \cdot 0 + 0.1 \cdot 4 = 0.4$ and the running variance $0.9 \cdot 1 + 0.1 \cdot 5 = 1.4$.
Show hint
For a BCHW tensor, which axes hold different observations of the same channel?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium