Gradient Descent Variants with MSE Loss

EasyML Coding

Problem statement

Implement one function gradient_descent(X, y, weights, learning_rate, n_epochs, batch_size=1, method='batch') that runs linear regression with Mean Squared Error loss using one of three gradient descent variants: 'batch', 'stochastic' or 'mini_batch'. Return the final weights.

The gradient of the MSE over a set $B$ of samples is

$$\nabla = \frac{2}{|B|} X_B^\top (X_B w - y_B).$$

Requirements

  • Do not shuffle; process samples in their original order.
  • Batch GD: one update per epoch using all samples.
  • Stochastic GD: one update per sample, in order 0, 1, 2, ...
  • Mini-batch GD: consecutive, non-overlapping batches (for batch_size=2: indices [0, 1], then [2, 3], ...). If the last batch is smaller, average its gradient over the samples it actually contains.
  • n_epochs is the number of full passes over the data.
  • Do not modify the input weights array in place.

Example

Input:

X = np.array([[1, 1], [2, 1], [3, 1], [4, 1]])
y = np.array([2, 3, 4, 5])
weights = np.zeros(2)
gradient_descent(X, y, weights, 0.01, 100, method='batch')

Output:

[1.14905239 0.56176776]

Reasoning: the data follows y = x + 1. After 100 full-batch updates with learning rate 0.01 the weights are moving toward [1, 1] but have not converged yet.

Show hint

What is the gradient of the mean squared error with respect to the weights for a linear model?

Relevant roles: ML Engineer, Data Scientist, Deep Learning Engineer and Applied Scientist.

Related free problems