Deriving Loss Gradients by Hand: Softmax Cross-Entropy

Reported in interviews at Bosch Fortinet Google

Problem statement

For a three-class classifier, let $z = Wx + b$, $p = \text{softmax}(z)$ and $L = -\log p_c$, where $c$ is the true class and $y$ is its one-hot label. Compute a stable loss, derive its gradient with respect to the logits, then pass that gradient back to the input.

Show hint

Write the loss in terms of the logits before differentiating.

Relevant roles: Deep Learning Engineer, ML Engineer and Research Scientist.

Related free problems