KL Divergence in LLM Training: Direction, Penalty and Estimation
Problem statement
How would you use KL divergence when post-training and distilling a chat model? Reinforcement learning keeps the policy near a frozen supervised reference, and distillation later trains a smaller student. The direction of the divergence changes what gets penalized, and sampled dashboard estimates need careful interpretation.
Calculate the divergence, choose a distillation direction, and interpret sampled estimates.
Show hint
Which distribution supplies the weights in the sum, and what happens to the log-ratio where one distribution is far smaller than the other?
Relevant roles: ML Engineer, Research Scientist, LLM Engineer and AI Engineer.
Related free problems
- Knowledge Distillation LossML CodingMedium