Linear Regression Using Gradient Descent
Problem statement
A baseline predictor estimates a continuous response from a fixed feature matrix. Implement batch gradient descent and then explain how you would diagnose training and select a checkpoint.
- Implement the judged function:
def linear_regression_gradient_descent(X, y, alpha, iterations):
...
X is a NumPy array of shape (m, n) whose first column is ones; y has shape (m,). The intercept is already included in the coefficient vector, so do not append a bias column. Start every coefficient at zero and minimize half the mean squared error, $L(\theta)=\frac{1}{2m}\sum_i((X\theta)_i-y_i)^2$, without regularization. Run exactly iterations simultaneous batch updates with learning rate alpha and return a one-dimensional NumPy array of shape (n,). Derive the gradient used in your implementation. If iterations is zero, return the initialized vector. Do not add early stopping to this fixed-update function. Judged outputs are compared after rounding to four decimal places.
- For a separate training workflow, specialize the gradients to one raw feature with prediction $wx+b$. Explain feature scaling, initialization, learning-rate selection, validation-based early stopping and how you would distinguish divergence, slow progress and overfitting. Do not change the judged function's update schedule.
Constraints
- Positive sample and feature counts, with matching finite floating-point arrays and a ones intercept column.
- Positive finite learning rate and a nonnegative integer update count.
- Use NumPy; do not use an optimizer or a regression-fitting library.
- Assume the requested updates remain representable in floating point.
Examples
X = np.array([[1., 0.], [1., 1.]])
y = np.array([1., 3.])
linear_regression_gradient_descent(X, y, 0.1, 1)
# [0.2, 0.15]
linear_regression_gradient_descent(X, y, 0.1, 0)
# [0., 0.]
The first call performs one update from zero; the second performs none.
Show hint
How does each residual depend on the coefficient vector?
Relevant roles: ML Engineer, Data Scientist, Applied Scientist and Deep Learning Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium