Deriving Loss Gradients by Hand: Softmax Cross-Entropy
Problem statement
For a three-class classifier, let $z = Wx + b$, $p = \text{softmax}(z)$ and $L = -\log p_c$, where $c$ is the true class and $y$ is its one-hot label. Compute a stable loss, derive its gradient with respect to the logits, then pass that gradient back to the input.
Show hint
Write the loss in terms of the logits before differentiating.
Relevant roles: Deep Learning Engineer, ML Engineer and Research Scientist.
Related free problems
- Derivative of Cross-Entropy Loss w.r.t. LogitsML CodingMedium
- Manual Backprop Through Cross-Entropy IntermediatesML CodingHard