Project · 13 steps · Hard

Neural Networks From Scratch: Forward and Backward

Implement a complete neural network stack from scratch in NumPy: finite-difference gradient checks, dense and activation layers, loss, sequential composition, optimizers, and a training loop that overfits then generalizes on a nonlinear dataset.

Each step is one function with tests. Your code runs in the browser, and every step builds on the ones before it.

Part 1: Gradient Verification Tools

Build central finite-difference numerical gradients and a relative-error checker used to validate every later analytic backward.

  1. numerical_gradient
  2. gradient_check

Part 2: Layers, Init, and Loss

Implement dense and nonlinear activation layers under a shared protocol, a symmetry-breaking initializer, and a numerically stable classification loss.

  1. make_dense
  2. make_activation
  3. initialize_weights
  4. make_loss

Part 3: Composition and End-to-End Pass

Compose layers into a sequential model and run a full forward-backward sweep whose parameter gradients match gradient_check.

  1. make_sequential
  2. forward_backward

Part 4: Optimization and Training Loop

Define an optimizer, a single train step that decreases loss, and a deterministic training loop that overfits a tiny batch.

  1. make_optimizer
  2. train_step
  3. train

Part 5: Architecture Design and Generalization

Design a network that solves a nonlinear dataset and improve held-out accuracy via regularization without collapsing to a constant predictor.

  1. design_network
  2. improve_generalization