Derivative of Cross-Entropy Loss w.r.t. Logits

MediumML Coding

Problem statement

Write a Python function cross_entropy_derivative(logits, target) that returns the gradient of the cross-entropy loss with respect to the input logits.

The logits $z$ are raw model scores. They pass through softmax, $p_i = e^{z_i} / \sum_j e^{z_j}$, and the loss for true class index target is $L = -\log p_{\text{target}}$. Return the vector $\partial L / \partial z$ as a list of floats with the same length as logits. The function should stay numerically stable for large logits.

Example

Input:

logits = [1.0, 2.0, 3.0]
target = 0

Output (rounded to 4 decimals):

[-0.91, 0.2447, 0.6652]

Reasoning: softmax gives $p \approx [0.09, 0.2447, 0.6652]$ and the one-hot target is $y = [1, 0, 0]$. The gradient is $p - y = [-0.91, 0.2447, 0.6652]$. The negative entry for class 0 says that raising that logit lowers the loss.

Show hint

What does the chain rule give when you push the loss derivative through the softmax Jacobian, and what cancels?

Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer, Research Scientist and Applied Scientist.

Related free problems