Backpropagation Through a Scalar Chain Network
Problem statement
Consider a feedforward neural network where every layer has exactly one neuron. The network has L layers with weights w = [w_1, ..., w_L] and biases b = [b_1, ..., b_L]. Given an input activation a0 and a target y, layer l computes
z_l = w_l * a_{l-1} + b_la_l = sigmoid(z_l)
and the cost for one example is C = (a_L - y) ** 2.
Implement backprop_chain(a0, y, w, b) that runs the forward pass and then uses the chain rule to compute the partial derivatives of C with respect to every weight and bias. Return a dictionary with two keys:
'dW': list[dC/dw_1, ..., dC/dw_L], each rounded to 6 decimals.'dB': list[dC/db_1, ..., dC/db_L], each rounded to 6 decimals.
Use $\sigma(z) = 1/(1 + e^{-z})$, whose derivative is $\sigma(z)(1 - \sigma(z))$.
Example
Input:
a0 = 1.0, y = 0.0, w = [1.0], b = [0.0]
Output:
{'dW': [0.28747], 'dB': [0.28747]}
Reasoning: forward, $z_1 = 1.0$ and $a_1 = \sigma(1) \approx 0.731059$. Backward, $\partial C/\partial a_1 = 2(0.731059 - 0) \approx 1.462117$ and $\sigma'(z_1) = a_1(1 - a_1) \approx 0.196612$, so $\partial C/\partial z_1 \approx 0.28747$. Then $\partial C/\partial w_1 = \partial C/\partial z_1 \cdot a_0 = 0.28747$ and $\partial C/\partial b_1 = 0.28747$.
Show hint
Which rule from calculus lets you get the derivative of a composition by multiplying local derivatives along the path?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium