Basic Autograd Operations
Problem statement
Implement a Value class that wraps a scalar and supports reverse-mode automatic differentiation. The class should:
- Support
+and*between twoValueobjects, or between aValue(on the left) and a plain number, returning a newValuewith the correct result. - Support a
relu()method that returns the input if it is positive and 0 otherwise. - Support a
backward()method that computes the gradient of the output with respect to every upstreamValuein the computation graph.
Each operation should record its inputs (_children) and a local rule for passing gradient back to them. backward() must process nodes in an order such that a node's gradient is complete before it is propagated further. The starter code provides __init__ and a __repr__ that prints Value(data=..., grad=...).
Example
Input:
a = Value(2)
b = Value(-3)
c = Value(10)
d = a + b * c
e = d.relu()
e.backward()
print(a, b, c, d, e)
Output:
Value(data=2, grad=0) Value(data=-3, grad=0) Value(data=10, grad=0) Value(data=-28, grad=0) Value(data=0, grad=1)
Reasoning: the forward pass gives $d = 2 + (-3)(10) = -28$, so the ReLU outputs 0. The ReLU derivative is 0 for a non-positive input, which blocks the gradient, so every node upstream of e ends with gradient 0.
Show hint
What does the chain rule say each primitive operation should pass back to its inputs, given the gradient of its output?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium