Variational Autoencoder Loss (Negative ELBO)
Problem statement
Implement the Variational Autoencoder (VAE) loss, the negative Evidence Lower Bound (ELBO). It has two parts:
- Reconstruction loss: for each sample, the sum of squared differences between the input and its reconstruction, averaged over the batch.
- KL divergence: how far the encoder's latent Gaussian $\mathcal{N}(\mu, \text{diag}(\sigma^2))$ is from the standard normal prior. The encoder outputs the mean
muand the log-variancelog_var. Per sample,
$$D_{KL} = -\frac{1}{2}\sum_{k}\left(1 + \text{log\_var}_k - \mu_k^2 - e^{\text{log\_var}_k}\right),$$ averaged over the batch.
Write vae_loss(x, x_reconstructed, mu, log_var) where x and x_reconstructed have shape (batch_size, features) and mu and log_var have shape (batch_size, latent_dim). Return a tuple of three floats (total_loss, reconstruction_loss, kl_divergence) with total_loss = reconstruction_loss + kl_divergence. Use only NumPy.
Example
Input:
x = np.array([[1.0, 0.0]])
x_reconstructed = np.array([[0.5, 0.5]])
mu = np.array([[1.0, 2.0]])
log_var = np.array([[0.0, 0.0]])
vae_loss(x, x_reconstructed, mu, log_var)
Output:
(3.0, 0.5, 2.5)
Reasoning: the reconstruction loss is $(1 - 0.5)^2 + (0 - 0.5)^2 = 0.5$. The KL term is $-\frac{1}{2}[(1 + 0 - 1 - 1) + (1 + 0 - 4 - 1)] = -\frac{1}{2}(-5) = 2.5$. The total is 3.0.
Show hint
What are the two competing goals in a VAE objective, and which one needs a distance between two probability distributions?
Relevant roles: Research Scientist, ML Engineer, Deep Learning Engineer, Gen AI Engineer and Applied Scientist.
Related free problems
- VAE Objective: ELBO Gap, Gaussian KL and Posterior CollapseDeep LearningMedium
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium