Project · 58 steps · Hard
Build a Tiny Autograd Engine from Scratch
Construct a minimal deep learning framework from scratch, layering a lazy numpy-backed buffer, a reverse-mode automatic differentiation engine, a tensor API, and neural network primitives. By the end you train a small MLP on a toy digit dataset, mirroring how real autograd libraries like tinygrad work under the hood.
Each step is one function with tests. Your code runs in the browser, and every step builds on the ones before it.
Part 1: Shape Utilities
Implement the small helper functions for working with shape tuples and index ordering that the rest of the engine relies on.
prodargsort
Part 2: The Lazy Buffer Backend
Build the numpy-backed LazyBuffer with constant and random initialization plus elementwise, reduce, and movement operations.
make_op_enumsLazyBufferlazybuffer_constrandlazybuffer_unary_elazybuffer_binary_elazybuffer_rlazybuffer_reshapelazybuffer_expandlazybuffer_permute
Part 3: Autograd Functions
Define the differentiable Function base class and implement forward and backward passes for all unary, binary, reduce, and movement operations.
Functionfunction_forward_backward_stubsapplyNegReluLogExpSqrtSigmoidAddSubMulDivsum_function_forwardsum_function_backwardmax_function_forwardmax_function_backwardReshapeexpand_function_forwardexpand_function_backwardpermute_function_forward_backward
Part 4: The Tensor API
Create the Tensor wrapper, creation helpers, topological backward pass, and method bindings for every supported operation.
Tensortensor_from_datatensor_creation_helperstensor_randnbuild_topological_ordertensor_backwardbind_unary_tensor_methodsbroadcastedbind_binary_tensor_methodsbind_movement_tensor_methodsbind_reduce_tensor_methods
Part 5: Functional and Composite Ops
Compose primitives into higher-level operations like mean, matmul, softmax, log_softmax, and cross entropy.
tensor_meantensor_transposetensor_matmul_2dtensor_softmaxtensor_log_softmaxsparse_categorical_cross_entropy
Part 6: Neural Network Modules and Optimizer
Implement Linear layers, an MLP, and the SGD optimizer with gradient zeroing.
LinearMLPsgd_stepzero_grad
Part 7: Training and Evaluation
Generate a toy dataset, train the MLP end to end, and evaluate its accuracy on a held-out split.
make_toy_digit_datasetaccuracytrain_mlpevaluate_mlp