Backpropagation Gradients for a Dense Layer
Problem statement
Implement backpropagation for a single fully connected layer with sigmoid activation and squared-error cost. The previous layer has K neurons (activations a_prev) and the current layer has J neurons.
The forward computation is
z = W @ a_prev + ba = sigmoid(z)C = sum_j (a_j - y_j)^2
Given a_prev (shape K), W (shape J x K), b (shape J) and target y (shape J), implement dense_backward(a_prev, W, b, y) that computes:
dC/dW(shapeJ x K)dC/db(shapeJ)dC/da_prev(shapeK). Eacha_prev[k]feeds allJneurons, so its gradient must aggregate contributions from every output neuron.
Return a dictionary with keys 'dW', 'db', 'da_prev' whose values are Python lists (nested for dW). Do not round inside the function.
Example
Input:
a_prev = [1.0, 2.0], W = [[0.1, 0.2], [0.3, 0.4]], b = [0.0, 0.0], y = [0.0, 1.0]
Output (rounded to 6 decimals):
{'dW': [[0.292561, 0.585121], [-0.093587, -0.187175]], 'db': [0.292561, -0.093587], 'da_prev': [0.00118, 0.021077]}
Reasoning: $z = Wa_{prev} + b = [0.5, 1.1]$ and $a = \sigma(z) \approx [0.6225, 0.7503]$. Then $\partial C/\partial z = 2(a - y)\,a(1 - a) \approx [0.2926, -0.0936]$, dW is its outer product with a_prev, db equals it, and da_prev $= W^\top \partial C/\partial z$ sums the contributions of both output neurons.
Show hint
What is the gradient of the cost with respect to each neuron's pre-activation, and which factors does the chain rule give you there?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist, Applied Scientist and Data Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium