Reparameterization Trick and the VAE Prior KL
Problem statement
A variational autoencoder's encoder outputs a mean $\mu$ and a log-variance $\log \sigma^2$ for each latent dimension. Sampling $z \sim \mathcal{N}(\mu, \sigma^2)$ directly would block gradients at the sampling step. The reparameterization trick moves the randomness outside the network:
$$z = \mu + \sigma \odot \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I), \qquad \sigma = \exp\!\big(\tfrac{1}{2} \log \sigma^2\big)$$
Implement two functions:
reparameterize(mu, logvar, eps):mu,logvar,epsall have shape(batch, latent_dim); returnzwith the same shape.epsis supplied so the result is deterministic.kl_to_standard_normal(mu, logvar): the KL divergence from $\mathcal{N}(\mu, \sigma^2)$ to $\mathcal{N}(0, I)$, summed over latent dimensions and averaged over the batch, as a float:
$$\mathrm{KL} = \frac{1}{B} \sum_{b=1}^{B} \sum_{j} \frac{1}{2}\big(\mu_{bj}^2 + \sigma_{bj}^2 - 1 - \log \sigma_{bj}^2\big)$$
Example
Input:
reparameterize(np.array([[1.0]]), np.array([[np.log(0.25)]]), np.array([[2.0]]))
kl_to_standard_normal(np.array([[1.0]]), np.array([[np.log(0.25)]]))
Output:
[[2.0]]
0.8181 (rounded to 4 decimals)
Reasoning: $\log \sigma^2 = \log 0.25$ gives $\sigma = 0.5$, so $z = 1 + 0.5 \cdot 2 = 2$. The KL is $\tfrac{1}{2}(1 + 0.25 - 1 + 1.3863) \approx 0.8181$.
Show hint
How can a random sample be rewritten so that the randomness does not depend on any learned quantity?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist, Applied Scientist and Gen AI Engineer.
Related free problems
- VAE Objective: ELBO Gap, Gaussian KL and Posterior CollapseDeep LearningMedium
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium