Derivative of Softmax

MediumML Coding

Problem statement

Write a Python function softmax_derivative(x) that computes the Jacobian matrix of the softmax function. Softmax maps a vector of real numbers to a probability distribution, and its derivative is what backpropagation needs when a classifier ends in a softmax layer.

Given an input vector x of length $n$, return the $n \times n$ Jacobian $J$ as a list of lists, where

$$J_{ij} = \frac{\partial \, \text{softmax}(x)_i}{\partial x_j}.$$

Your implementation should stay numerically stable for large inputs.

Example

Input:

x = [1.0, 2.0, 3.0]

Output (rounded to 4 decimals):

[[0.0819, -0.022, -0.0599], [-0.022, 0.1848, -0.1628], [-0.0599, -0.1628, 0.2227]]

Reasoning: softmax gives $s \approx [0.09, 0.2447, 0.6652]$. Diagonal entries are $s_i(1 - s_i)$, so $J_{00} = 0.09 \cdot 0.91 \approx 0.0819$. Off-diagonal entries are $-s_i s_j$, so $J_{01} = -0.09 \cdot 0.2447 \approx -0.022$.

Show hint

Which differentiation rule applies when every output is a ratio whose denominator depends on all the inputs?

Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.

Related free problems