KL Divergence Between Two Multivariate Gaussian Distributions
Problem statement
The Kullback-Leibler (KL) divergence measures how one probability distribution differs from another. Implement multivariate_kl_divergence(mu_p, Cov_p, mu_q, Cov_q) that returns $D_{KL}(P \,\|\, Q)$ for two multivariate Gaussians $P = \mathcal{N}(\mu_p, \Sigma_p)$ and $Q = \mathcal{N}(\mu_q, \Sigma_q)$ in $k$ dimensions:
$$D_{KL}(P \,\|\, Q) = \frac{1}{2}\left[\log\frac{\det \Sigma_q}{\det \Sigma_p} - k + \mathrm{tr}(\Sigma_q^{-1}\Sigma_p) + (\mu_q - \mu_p)^\top \Sigma_q^{-1} (\mu_q - \mu_p)\right].$$
mu_p,mu_q: NumPy arrays of shape $(k,)$.Cov_p,Cov_q: symmetric positive definite NumPy arrays of shape $(k, k)$.
Return the divergence as a Python float.
Example
Input:
mu_p = np.array([1.0, 2.0]), Cov_p = np.eye(2)
mu_q = np.array([0.0, 0.0]), Cov_q = 2 * np.eye(2)
Output (rounded to 6 decimals):
1.443147
Reasoning: the log-determinant term is $\log(4/1) = 1.386294$, $k = 2$, the trace term is $\mathrm{tr}(\tfrac{1}{2} I) = 1$, and the Mahalanobis term is $(1^2 + 2^2)/2 = 2.5$. So the KL is $\tfrac{1}{2}(1.386294 - 2 + 1 + 2.5) = 1.443147$.
Show hint
Which parts of the divergence come from the difference in means and which from the difference in covariances?
Relevant roles: Research Scientist, ML Engineer, Applied Scientist and Deep Learning Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium