Gradient Descent Variants with MSE Loss
Problem statement
Implement one function gradient_descent(X, y, weights, learning_rate, n_epochs, batch_size=1, method='batch') that runs linear regression with Mean Squared Error loss using one of three gradient descent variants: 'batch', 'stochastic' or 'mini_batch'. Return the final weights.
The gradient of the MSE over a set $B$ of samples is
$$\nabla = \frac{2}{|B|} X_B^\top (X_B w - y_B).$$
Requirements
- Do not shuffle; process samples in their original order.
- Batch GD: one update per epoch using all samples.
- Stochastic GD: one update per sample, in order 0, 1, 2, ...
- Mini-batch GD: consecutive, non-overlapping batches (for
batch_size=2: indices [0, 1], then [2, 3], ...). If the last batch is smaller, average its gradient over the samples it actually contains. n_epochsis the number of full passes over the data.- Do not modify the input
weightsarray in place.
Example
Input:
X = np.array([[1, 1], [2, 1], [3, 1], [4, 1]])
y = np.array([2, 3, 4, 5])
weights = np.zeros(2)
gradient_descent(X, y, weights, 0.01, 100, method='batch')
Output:
[1.14905239 0.56176776]
Reasoning: the data follows y = x + 1. After 100 full-batch updates with learning rate 0.01 the weights are moving toward [1, 1] but have not converged yet.
Show hint
What is the gradient of the mean squared error with respect to the weights for a linear model?
Relevant roles: ML Engineer, Data Scientist, Deep Learning Engineer and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium