Neural networks

Multilayer perceptron

Imagine a team passing notes: each person combines the notes they receive before passing a new one along. An MLP stacks layers of units that mix input features with learned weights. Nonlinear activations let the network learn more than a straight-line rule.

Input2 featuresHidden layer3 ReLU unitsOutput1 score
A 2-to-3-to-1 MLP uses a 3 × 2 first weight matrix and a 1 × 3 output weight matrix.
Input2 featuresHidden layer3 ReLU unitsOutput1 score
A 2-to-3-to-1 MLP uses a 3 × 2 first weight matrix and a 1 × 3 output weight matrix.

How it works

  1. Multiply the input by the first weight matrix and add a bias.
  2. Apply a nonlinear activation such as ReLU to form hidden features.
  3. Repeat the layer calculation and map the final scores to predictions appropriate for the task.
  4. Compute a loss, backpropagate its gradients and update the weights with an optimizer.

$$h=\operatorname{ReLU}(W_1x+b_1),\qquad z=W_2h+b_2$$

The knobs that matter

hidden width
more units increase capacity and computation
depth
more layers compose more transformations but can be harder to train
activation
sets the nonlinear transformation between linear layers
weight decay
shrinks weights during updates to help control overfitting

What interviewers ask

  • When would you choose an MLP over a CNN or an RNN?
  • Why do we need nonlinear activations between layers?
  • Can you work out the shapes and gradients for a two-layer network?

Common mistakes

  • Stacking linear layers without nonlinear activations still gives one affine transformation.
  • Applying softmax before a loss that already expects logits can produce the wrong training calculation.
  • Ignoring feature scales can make optimization unnecessarily difficult.

When to use it

Use an MLP for fixed-size feature vectors or as a learned transformation inside a larger network. For images, graphs or sequences, consider architectures that directly use their structure.

Implement it (4 problems)

Build it step by step

Learn it

Interview questions about multilayer perceptron