Neural networks
Multilayer perceptron
Imagine a team passing notes: each person combines the notes they receive before passing a new one along. An MLP stacks layers of units that mix input features with learned weights. Nonlinear activations let the network learn more than a straight-line rule.
How it works
- Multiply the input by the first weight matrix and add a bias.
- Apply a nonlinear activation such as ReLU to form hidden features.
- Repeat the layer calculation and map the final scores to predictions appropriate for the task.
- Compute a loss, backpropagate its gradients and update the weights with an optimizer.
$$h=\operatorname{ReLU}(W_1x+b_1),\qquad z=W_2h+b_2$$
The knobs that matter
- hidden width
- more units increase capacity and computation
- depth
- more layers compose more transformations but can be harder to train
- activation
- sets the nonlinear transformation between linear layers
- weight decay
- shrinks weights during updates to help control overfitting
What interviewers ask
- When would you choose an MLP over a CNN or an RNN?
- Why do we need nonlinear activations between layers?
- Can you work out the shapes and gradients for a two-layer network?
Common mistakes
- Stacking linear layers without nonlinear activations still gives one affine transformation.
- Applying softmax before a loss that already expects logits can produce the wrong training calculation.
- Ignoring feature scales can make optimization unnecessarily difficult.
When to use it
Use an MLP for fixed-size feature vectors or as a learned transformation inside a larger network. For images, graphs or sequences, consider architectures that directly use their structure.
Implement it (4 problems)
Build it step by step
- Neural Networks From Scratch: Forward and Backward13 steps, your code runs in the browser
- Build a Tiny Autograd Engine from Scratch58 steps, your code runs in the browser
Learn it
Interview questions about multilayer perceptron
- MLP vs CNN vs RNNProEasySolve