Derivative of Softmax
Problem statement
Write a Python function softmax_derivative(x) that computes the Jacobian matrix of the softmax function. Softmax maps a vector of real numbers to a probability distribution, and its derivative is what backpropagation needs when a classifier ends in a softmax layer.
Given an input vector x of length $n$, return the $n \times n$ Jacobian $J$ as a list of lists, where
$$J_{ij} = \frac{\partial \, \text{softmax}(x)_i}{\partial x_j}.$$
Your implementation should stay numerically stable for large inputs.
Example
Input:
x = [1.0, 2.0, 3.0]
Output (rounded to 4 decimals):
[[0.0819, -0.022, -0.0599], [-0.022, 0.1848, -0.1628], [-0.0599, -0.1628, 0.2227]]
Reasoning: softmax gives $s \approx [0.09, 0.2447, 0.6652]$. Diagonal entries are $s_i(1 - s_i)$, so $J_{00} = 0.09 \cdot 0.91 \approx 0.0819$. Off-diagonal entries are $-s_i s_j$, so $J_{01} = -0.09 \cdot 0.2447 \approx -0.022$.
Show hint
Which differentiation rule applies when every output is a ratio whose denominator depends on all the inputs?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium