Logistic Regression: Gradient, Hessian, and Convexity

Reported in interviews at Bytedance Snap Two Sigma Yandex Stripe Amazon

Problem statement

Consider a binary classification problem with $n$ observations. Each observation has a label $y_i \in \{0, 1\}$ and a feature vector $\mathbf{x}_i \in \mathbb{R}^d$. The logistic regression model with intercept models the probability of $y_i = 1$ as:

$$p_i(\boldsymbol{\theta}) = \sigma(\boldsymbol{\theta}^\top \tilde{\mathbf{x}}_i) = \frac{1}{1 + e^{-\boldsymbol{\theta}^\top \tilde{\mathbf{x}}_i}}$$

where $\tilde{\mathbf{x}}_i = (1, \mathbf{x}_i^\top)^\top \in \mathbb{R}^{d+1}$ is the augmented feature vector (prepending a 1 for the intercept) and $\boldsymbol{\theta} \in \mathbb{R}^{d+1}$ is the parameter vector.

  1. Write down the log-likelihood $\ell(\boldsymbol{\theta})$ for the full dataset.
  1. Derive the gradient $\nabla \ell(\boldsymbol{\theta})$.
  1. Derive the Hessian $\mathbf{H}(\boldsymbol{\theta})$.
  1. Show that $-\ell(\boldsymbol{\theta})$ is convex in $\boldsymbol{\theta}$.
  1. Explain why Newton's method (equivalently, IRLS) converges to the global maximum under standard regularity conditions.
Show hint

What derivative identity of the sigmoid makes the log-likelihood easy to differentiate?

Relevant roles: ML Engineer, Applied Scientist, Research Scientist and Data Scientist.

Related free problems