ML Fundamentals Round: Cross-Entropy, Dropout, Normalization, Adam and PPO

MediumDeep Learning

Reported in interviews at Snap

Problem statement

How would you reason through a few common model training mistakes? We will start with a classifier trained on annotators' soft labels, then examine a ranking model whose dropout changes score scale at serving. Finally, we will consider normalization in a language model that generates tokens incrementally.

Calculate the reducible loss, fix dropout scaling and explain the normalization choice.

Show hint

For each question, ask what the definition implies about a concrete quantity: a loss floor, an activation scale, the axis statistics are taken over.

Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.

Related free problems