KL Divergence in LLM Training: Direction, Penalty and Estimation

HardLLMs & GenAI

Reported in interviews at Google

Problem statement

How would you use KL divergence when post-training and distilling a chat model? Reinforcement learning keeps the policy near a frozen supervised reference, and distillation later trains a smaller student. The direction of the divergence changes what gets penalized, and sampled dashboard estimates need careful interpretation.

Calculate the divergence, choose a distillation direction, and interpret sampled estimates.

Show hint

Which distribution supplies the weights in the sum, and what happens to the log-ratio where one distribution is far smaller than the other?

Relevant roles: ML Engineer, Research Scientist, LLM Engineer and AI Engineer.

Related free problems