A Network Too Sensitive to Its Initialization; Batch Norm and Dropout at Test Time

Reported in interviews at DRW GSA Capital

Problem statement

(a) Your neural network's results swing wildly depending on its initial weights. What do you do?

(b) How do batch normalization and dropout each behave differently at test time than during training?

Show hint

What mechanisms make a deep network amplify small differences in its starting weights, and which parts of the training setup control them?

Relevant roles: Deep Learning Engineer, LLM Engineer, ML Engineer and Research Scientist.

Related free problems