KL Divergence Between Two Multivariate Gaussian Distributions

EasyML Coding

Problem statement

The Kullback-Leibler (KL) divergence measures how one probability distribution differs from another. Implement multivariate_kl_divergence(mu_p, Cov_p, mu_q, Cov_q) that returns $D_{KL}(P \,\|\, Q)$ for two multivariate Gaussians $P = \mathcal{N}(\mu_p, \Sigma_p)$ and $Q = \mathcal{N}(\mu_q, \Sigma_q)$ in $k$ dimensions:

$$D_{KL}(P \,\|\, Q) = \frac{1}{2}\left[\log\frac{\det \Sigma_q}{\det \Sigma_p} - k + \mathrm{tr}(\Sigma_q^{-1}\Sigma_p) + (\mu_q - \mu_p)^\top \Sigma_q^{-1} (\mu_q - \mu_p)\right].$$

  • mu_p, mu_q: NumPy arrays of shape $(k,)$.
  • Cov_p, Cov_q: symmetric positive definite NumPy arrays of shape $(k, k)$.

Return the divergence as a Python float.

Example

Input:

mu_p = np.array([1.0, 2.0]), Cov_p = np.eye(2)
mu_q = np.array([0.0, 0.0]), Cov_q = 2 * np.eye(2)

Output (rounded to 6 decimals):

1.443147

Reasoning: the log-determinant term is $\log(4/1) = 1.386294$, $k = 2$, the trace term is $\mathrm{tr}(\tfrac{1}{2} I) = 1$, and the Mahalanobis term is $(1^2 + 2^2)/2 = 2.5$. So the KL is $\tfrac{1}{2}(1.386294 - 2 + 1 + 2.5) = 1.443147$.

Show hint

Which parts of the divergence come from the difference in means and which from the difference in covariances?

Relevant roles: Research Scientist, ML Engineer, Applied Scientist and Deep Learning Engineer.

Related free problems