Linear Regression Using Gradient Descent

MediumML Coding

Reported in interviews at Databricks General Motors Indeed

Problem statement

A baseline predictor estimates a continuous response from a fixed feature matrix. Implement batch gradient descent and then explain how you would diagnose training and select a checkpoint.

  1. Implement the judged function:
def linear_regression_gradient_descent(X, y, alpha, iterations):
    ...

X is a NumPy array of shape (m, n) whose first column is ones; y has shape (m,). The intercept is already included in the coefficient vector, so do not append a bias column. Start every coefficient at zero and minimize half the mean squared error, $L(\theta)=\frac{1}{2m}\sum_i((X\theta)_i-y_i)^2$, without regularization. Run exactly iterations simultaneous batch updates with learning rate alpha and return a one-dimensional NumPy array of shape (n,). Derive the gradient used in your implementation. If iterations is zero, return the initialized vector. Do not add early stopping to this fixed-update function. Judged outputs are compared after rounding to four decimal places.

  1. For a separate training workflow, specialize the gradients to one raw feature with prediction $wx+b$. Explain feature scaling, initialization, learning-rate selection, validation-based early stopping and how you would distinguish divergence, slow progress and overfitting. Do not change the judged function's update schedule.

Constraints

  • Positive sample and feature counts, with matching finite floating-point arrays and a ones intercept column.
  • Positive finite learning rate and a nonnegative integer update count.
  • Use NumPy; do not use an optimizer or a regression-fitting library.
  • Assume the requested updates remain representable in floating point.

Examples

X = np.array([[1., 0.], [1., 1.]])
y = np.array([1., 3.])
linear_regression_gradient_descent(X, y, 0.1, 1)
# [0.2, 0.15]
linear_regression_gradient_descent(X, y, 0.1, 0)
# [0., 0.]

The first call performs one update from zero; the second performs none.

Show hint

How does each residual depend on the coefficient vector?

Relevant roles: ML Engineer, Data Scientist, Applied Scientist and Deep Learning Engineer.

Related free problems