Basic Autograd Operations

MediumML Coding

Problem statement

Implement a Value class that wraps a scalar and supports reverse-mode automatic differentiation. The class should:

  1. Support + and * between two Value objects, or between a Value (on the left) and a plain number, returning a new Value with the correct result.
  2. Support a relu() method that returns the input if it is positive and 0 otherwise.
  3. Support a backward() method that computes the gradient of the output with respect to every upstream Value in the computation graph.

Each operation should record its inputs (_children) and a local rule for passing gradient back to them. backward() must process nodes in an order such that a node's gradient is complete before it is propagated further. The starter code provides __init__ and a __repr__ that prints Value(data=..., grad=...).

Example

Input:

a = Value(2)
b = Value(-3)
c = Value(10)
d = a + b * c
e = d.relu()
e.backward()
print(a, b, c, d, e)

Output:

Value(data=2, grad=0) Value(data=-3, grad=0) Value(data=10, grad=0) Value(data=-28, grad=0) Value(data=0, grad=1)

Reasoning: the forward pass gives $d = 2 + (-3)(10) = -28$, so the ReLU outputs 0. The ReLU derivative is 0 for a non-positive input, which blocks the gradient, so every node upstream of e ends with gradient 0.

Show hint

What does the chain rule say each primitive operation should pass back to its inputs, given the gradient of its output?

Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.

Related free problems