Knowledge Distillation Loss
Problem statement
Implement the knowledge distillation loss used to train a small student model to imitate a larger teacher. Instead of hard labels, the student matches the teacher's temperature-softened probability distribution:
$$L = T^2 \cdot \mathrm{KL}\big(\,\text{softmax}(z_t / T) \;\|\; \text{softmax}(z_s / T)\,\big) = T^2 \sum_i p_i \log \frac{p_i}{q_i},$$
where $p = \text{softmax}(z_t / T)$ is the teacher distribution, $q = \text{softmax}(z_s / T)$ is the student distribution, and $T$ is the temperature.
Write distillation_loss(student_logits, teacher_logits, temperature=1.0) for 1D logit arrays and return the loss as a float. Make the softmax numerically stable.
Example
Input:
student_logits = np.array([2.0, -2.0]), teacher_logits = np.array([1.5, -1.5]), temperature = 1.0
Output:
0.017
Reasoning: the teacher is slightly less confident than the student (logit gap 3.0 versus 4.0), so the two distributions differ a little and the KL divergence is small but positive. Only logit differences matter, since softmax is shift-invariant.
Show hint
What happens to a softmax distribution when you divide the logits by a temperature larger than one, and why would that help a student learn?
Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.
Related free problems
- KL Divergence in LLM Training: Direction, Penalty and EstimationLLMs & GenAIHard
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium