Project · 13 steps · Hard
Neural Networks From Scratch: Forward and Backward
Implement a complete neural network stack from scratch in NumPy: finite-difference gradient checks, dense and activation layers, loss, sequential composition, optimizers, and a training loop that overfits then generalizes on a nonlinear dataset.
Each step is one function with tests. Your code runs in the browser, and every step builds on the ones before it.
Part 1: Gradient Verification Tools
Build central finite-difference numerical gradients and a relative-error checker used to validate every later analytic backward.
numerical_gradientgradient_check
Part 2: Layers, Init, and Loss
Implement dense and nonlinear activation layers under a shared protocol, a symmetry-breaking initializer, and a numerically stable classification loss.
make_densemake_activationinitialize_weightsmake_loss
Part 3: Composition and End-to-End Pass
Compose layers into a sequential model and run a full forward-backward sweep whose parameter gradients match gradient_check.
make_sequentialforward_backward
Part 4: Optimization and Training Loop
Define an optimizer, a single train step that decreases loss, and a deterministic training loop that overfits a tiny batch.
make_optimizertrain_steptrain
Part 5: Architecture Design and Generalization
Design a network that solves a nonlinear dataset and improve held-out accuracy via regularization without collapsing to a constant predictor.
design_networkimprove_generalization