Knowledge Distillation Loss

MediumML Coding

Problem statement

Implement the knowledge distillation loss used to train a small student model to imitate a larger teacher. Instead of hard labels, the student matches the teacher's temperature-softened probability distribution:

$$L = T^2 \cdot \mathrm{KL}\big(\,\text{softmax}(z_t / T) \;\|\; \text{softmax}(z_s / T)\,\big) = T^2 \sum_i p_i \log \frac{p_i}{q_i},$$

where $p = \text{softmax}(z_t / T)$ is the teacher distribution, $q = \text{softmax}(z_s / T)$ is the student distribution, and $T$ is the temperature.

Write distillation_loss(student_logits, teacher_logits, temperature=1.0) for 1D logit arrays and return the loss as a float. Make the softmax numerically stable.

Example

Input:

student_logits = np.array([2.0, -2.0]), teacher_logits = np.array([1.5, -1.5]), temperature = 1.0

Output:

0.017

Reasoning: the teacher is slightly less confident than the student (logit gap 3.0 versus 4.0), so the two distributions differ a little and the KL divergence is small but positive. Only logit differences matter, since softmax is shift-invariant.

Show hint

What happens to a softmax distribution when you divide the logits by a temperature larger than one, and why would that help a student learn?

Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.

Related free problems