Manual Backprop Through Cross-Entropy Intermediates
Problem statement
Implement the manual backward pass through every intermediate tensor of the softmax + negative-log-likelihood loss, without relying on any automatic differentiation engine.
Given logits and integer class targets, the forward pass is decomposed into the following intermediates (in this exact order):
logit_maxes = logits.max(axis=1, keepdims=True)(shape(n, 1))norm_logits = logits - logit_maxes(shape(n, C))counts = exp(norm_logits)(shape(n, C))counts_sum = counts.sum(axis=1, keepdims=True)(shape(n, 1))counts_sum_inv = counts_sum ** -1(shape(n, 1))probs = counts * counts_sum_inv(shape(n, C))logprobs = log(probs)(shape(n, C))loss = -mean(logprobs[range(n), y])(scalar)
Your function must compute the gradient of loss with respect to each of these intermediates, in the same shape as the corresponding forward tensor. The gradient w.r.t. logits must combine both paths (the direct logits - logit_maxes subtraction and the logits.max reduction).
Inputs
logits: numpy array of shape(n, C)y: numpy array of integer class indices, shape(n,)
Output
Return a dictionary with these eight keys, each containing a numpy array with the shape of the corresponding forward intermediate:
'dlogprobs','dprobs','dcounts_sum_inv','dcounts_sum','dcounts','dnorm_logits','dlogit_maxes','dlogits'
Notes
- The mean in the loss is over the batch dimension
n, so the seed gradient at each picked logprob entry is-1/n. - The gradient through
counts.sum(axis=1, keepdims=True)broadcasts back to every column, so anones_like(counts)factor is needed. - The gradient through
logits.max(axis=1, keepdims=True)is a one-hot at the argmax index of each row. dcountsreceives contributions from two paths (throughprobsand throughcounts_sum) and must be the sum of both.- Do not round inside your solution.
Example
Input:
logits = np.array([[0.0, 0.0]])
y = np.array([0])
Output:
{'dlogprobs': [[-1.0, 0.0]], 'dprobs': [[-2.0, 0.0]], 'dcounts_sum_inv': [[-2.0]], 'dcounts_sum': [[0.5]], 'dcounts': [[-0.5, 0.5]], 'dnorm_logits': [[-0.5, 0.5]], 'dlogit_maxes': [[0.0]], 'dlogits': [[-0.5, 0.5]]}
Reasoning: probs = [0.5, 0.5]. The seed at the picked column is -1/n = -1, so dlogprobs = [[-1, 0]] and dprobs = dlogprobs / probs = [[-2, 0]]. dcounts_sum_inv = sum(counts dprobs) = -2. dcounts_sum = -counts_sum^-2 (-2) = -(1/4)(-2) = 0.5. dcounts = 0.5 [[-2, 0]] + 0.5 = [[-0.5, 0.5]], dnorm_logits = counts dcounts = [[-0.5, 0.5]], dlogit_maxes = -sum(dnorm_logits) = 0, and dlogits = [[-0.5, 0.5]] = probs - one_hot(y).
Show hint
Starting from the loss, which single entry per row of the log-probabilities actually receives gradient?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.
Related free problems
- Deriving Loss Gradients by Hand: Softmax Cross-EntropyDeep LearningEasy
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium