Variational Autoencoder Loss (Negative ELBO)

MediumML Coding

Problem statement

Implement the Variational Autoencoder (VAE) loss, the negative Evidence Lower Bound (ELBO). It has two parts:

  1. Reconstruction loss: for each sample, the sum of squared differences between the input and its reconstruction, averaged over the batch.
  2. KL divergence: how far the encoder's latent Gaussian $\mathcal{N}(\mu, \text{diag}(\sigma^2))$ is from the standard normal prior. The encoder outputs the mean mu and the log-variance log_var. Per sample,

$$D_{KL} = -\frac{1}{2}\sum_{k}\left(1 + \text{log\_var}_k - \mu_k^2 - e^{\text{log\_var}_k}\right),$$ averaged over the batch.

Write vae_loss(x, x_reconstructed, mu, log_var) where x and x_reconstructed have shape (batch_size, features) and mu and log_var have shape (batch_size, latent_dim). Return a tuple of three floats (total_loss, reconstruction_loss, kl_divergence) with total_loss = reconstruction_loss + kl_divergence. Use only NumPy.

Example

Input:

x = np.array([[1.0, 0.0]])
x_reconstructed = np.array([[0.5, 0.5]])
mu = np.array([[1.0, 2.0]])
log_var = np.array([[0.0, 0.0]])
vae_loss(x, x_reconstructed, mu, log_var)

Output:

(3.0, 0.5, 2.5)

Reasoning: the reconstruction loss is $(1 - 0.5)^2 + (0 - 0.5)^2 = 0.5$. The KL term is $-\frac{1}{2}[(1 + 0 - 1 - 1) + (1 + 0 - 4 - 1)] = -\frac{1}{2}(-5) = 2.5$. The total is 3.0.

Show hint

What are the two competing goals in a VAE objective, and which one needs a distance between two probability distributions?

Relevant roles: Research Scientist, ML Engineer, Deep Learning Engineer, Gen AI Engineer and Applied Scientist.

Related free problems