Neural networks

Dropout and normalization

Imagine practicing a group project with a few teammates absent, and comparing measurements taken with different rulers. Dropout randomly zeros activations during training to reduce reliance on particular units. Normalization centers and rescales groups of activations using their statistics; the groups depend on the method.

For input [1, 2, 3, 4] and dropout probability 0.5, kept activations are doubled during training.

How it works

  1. During training, inverted dropout samples a mask and scales kept activations by $1/(1-p)$ for dropout probability $0\leq p<1$.
  2. For normalization, compute statistics across the axes specified by the method and rescale with a small numerical stabilizer.
  3. Where supported, apply learned scale and shift parameters after normalization.
  4. At inference, disable standard dropout and use stored running statistics for standard batch normalization, while layer normalization still uses the current input.

$$y=\gamma\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta$$

The knobs that matter

dropout probability
higher values remove more activations during training
normalization axes
batch normalization shares channel statistics across examples; layer normalization uses selected features within each example
epsilon
prevents division by very small scales
running-statistic update rate
controls how quickly batch normalization tracks training statistics

What interviewers ask

  • What changes for dropout and batch normalization at inference time?
  • Which axes do batch normalization and layer normalization normalize?
  • Why can batch normalization behave poorly with tiny batches?
  • How does RMS normalization differ from layer normalization?

Common mistakes

  • Leaving the model in training mode can keep dropout active and change batch normalization behavior at inference.
  • Using the wrong axes changes which activations share statistics.
  • RMS normalization divides by root mean square without subtracting the mean, so it differs from the centered formula shown here.

When to use it

Use dropout when validation results show it helps regularization, and choose normalization to match the architecture and batch setup. Neither replaces correct data splits or guarantees that a model will generalize.

Implement it (5 problems)

Learn it

Interview questions about dropout and normalization