Neural networks

Autoencoder and VAE

Imagine squeezing a photo into a short description and asking someone to redraw it from that description. An autoencoder learns an encoder and decoder to reconstruct inputs through a latent code. A variational autoencoder (VAE) learns a distribution over codes and regularizes it toward a prior so new codes can be sampled.

Input4 featuresLatent code2 valuesReconstruction4 features
A 4-to-2-to-4 bottleneck forces reconstruction through a smaller code; a VAE encoder instead outputs distribution parameters.
Input4 featuresLatent code2 valuesReconstruction4 features
A 4-to-2-to-4 bottleneck forces reconstruction through a smaller code; a VAE encoder instead outputs distribution parameters.

How it works

  1. Encode each input into a code, or into the mean and variance of a Gaussian VAE’s approximate posterior over latent codes.
  2. For a Gaussian VAE, sample with $z=\mu+\sigma\odot\epsilon$, where $\epsilon\sim\mathcal{N}(0,I)$, so gradients can reach the encoder.
  3. Decode the code into a reconstruction or the parameters of an observation distribution.
  4. Minimize reconstruction error for an autoencoder; for a VAE, minimize expected negative log likelihood plus a KL penalty toward the prior.

$$\mathcal{L}_{\mathrm{VAE}}=-\mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]+D_{\mathrm{KL}}(q_\phi(z\mid x)\Vert p(z))$$

The knobs that matter

latent dimension
sets code capacity and the amount of compression
reconstruction likelihood
defines which reconstruction errors the model penalizes
KL weight
trades reconstruction against prior matching in a weighted VAE objective
encoder and decoder capacity
controls how much structure each side can represent

What interviewers ask

  • How does a VAE differ from a regular autoencoder?
  • Why do we use the reparameterization trick?
  • What is the ELBO, and how does it relate to log likelihood?
  • What is posterior collapse, and how would you detect it?

Common mistakes

  • A plain autoencoder does not ensure that arbitrary latent samples decode into useful outputs.
  • Mixing sums and means across dimensions can unintentionally change the balance of reconstruction and KL terms.
  • A very small KL term can mean the decoder is ignoring the latent code rather than learning a useful representation.

When to use it

Use autoencoders for learned representations, reconstruction or denoising, and VAEs when you need a probabilistic latent model. Reconstruction quality alone does not establish that latent features are useful or that anomaly scores are reliable.

Implement it (7 problems)

Learn it

Interview questions about autoencoder and vae