Logistic Regression: Gradient, Hessian, and Convexity
Problem statement
Consider a binary classification problem with $n$ observations. Each observation has a label $y_i \in \{0, 1\}$ and a feature vector $\mathbf{x}_i \in \mathbb{R}^d$. The logistic regression model with intercept models the probability of $y_i = 1$ as:
$$p_i(\boldsymbol{\theta}) = \sigma(\boldsymbol{\theta}^\top \tilde{\mathbf{x}}_i) = \frac{1}{1 + e^{-\boldsymbol{\theta}^\top \tilde{\mathbf{x}}_i}}$$
where $\tilde{\mathbf{x}}_i = (1, \mathbf{x}_i^\top)^\top \in \mathbb{R}^{d+1}$ is the augmented feature vector (prepending a 1 for the intercept) and $\boldsymbol{\theta} \in \mathbb{R}^{d+1}$ is the parameter vector.
- Write down the log-likelihood $\ell(\boldsymbol{\theta})$ for the full dataset.
- Derive the gradient $\nabla \ell(\boldsymbol{\theta})$.
- Derive the Hessian $\mathbf{H}(\boldsymbol{\theta})$.
- Show that $-\ell(\boldsymbol{\theta})$ is convex in $\boldsymbol{\theta}$.
- Explain why Newton's method (equivalently, IRLS) converges to the global maximum under standard regularity conditions.
Show hint
What derivative identity of the sigmoid makes the log-likelihood easy to differentiate?
Relevant roles: ML Engineer, Applied Scientist, Research Scientist and Data Scientist.