Project · 58 steps · Hard

Build a Tiny Autograd Engine from Scratch

Construct a minimal deep learning framework from scratch, layering a lazy numpy-backed buffer, a reverse-mode automatic differentiation engine, a tensor API, and neural network primitives. By the end you train a small MLP on a toy digit dataset, mirroring how real autograd libraries like tinygrad work under the hood.

Each step is one function with tests. Your code runs in the browser, and every step builds on the ones before it.

Part 1: Shape Utilities

Implement the small helper functions for working with shape tuples and index ordering that the rest of the engine relies on.

  1. prod
  2. argsort

Part 2: The Lazy Buffer Backend

Build the numpy-backed LazyBuffer with constant and random initialization plus elementwise, reduce, and movement operations.

  1. make_op_enums
  2. LazyBuffer
  3. lazybuffer_const
  4. rand
  5. lazybuffer_unary_e
  6. lazybuffer_binary_e
  7. lazybuffer_r
  8. lazybuffer_reshape
  9. lazybuffer_expand
  10. lazybuffer_permute

Part 3: Autograd Functions

Define the differentiable Function base class and implement forward and backward passes for all unary, binary, reduce, and movement operations.

  1. Function
  2. function_forward_backward_stubs
  3. apply
  4. Neg
  5. Relu
  6. Log
  7. Exp
  8. Sqrt
  9. Sigmoid
  10. Add
  11. Sub
  12. Mul
  13. Div
  14. sum_function_forward
  15. sum_function_backward
  16. max_function_forward
  17. max_function_backward
  18. Reshape
  19. expand_function_forward
  20. expand_function_backward
  21. permute_function_forward_backward

Part 4: The Tensor API

Create the Tensor wrapper, creation helpers, topological backward pass, and method bindings for every supported operation.

  1. Tensor
  2. tensor_from_data
  3. tensor_creation_helpers
  4. tensor_randn
  5. build_topological_order
  6. tensor_backward
  7. bind_unary_tensor_methods
  8. broadcasted
  9. bind_binary_tensor_methods
  10. bind_movement_tensor_methods
  11. bind_reduce_tensor_methods

Part 5: Functional and Composite Ops

Compose primitives into higher-level operations like mean, matmul, softmax, log_softmax, and cross entropy.

  1. tensor_mean
  2. tensor_transpose
  3. tensor_matmul_2d
  4. tensor_softmax
  5. tensor_log_softmax
  6. sparse_categorical_cross_entropy

Part 6: Neural Network Modules and Optimizer

Implement Linear layers, an MLP, and the SGD optimizer with gradient zeroing.

  1. Linear
  2. MLP
  3. sgd_step
  4. zero_grad

Part 7: Training and Evaluation

Generate a toy dataset, train the MLP end to end, and evaluate its accuracy on a held-out split.

  1. make_toy_digit_dataset
  2. accuracy
  3. train_mlp
  4. evaluate_mlp