Neural networks
Autoencoder and VAE
Imagine squeezing a photo into a short description and asking someone to redraw it from that description. An autoencoder learns an encoder and decoder to reconstruct inputs through a latent code. A variational autoencoder (VAE) learns a distribution over codes and regularizes it toward a prior so new codes can be sampled.
How it works
- Encode each input into a code, or into the mean and variance of a Gaussian VAE’s approximate posterior over latent codes.
- For a Gaussian VAE, sample with $z=\mu+\sigma\odot\epsilon$, where $\epsilon\sim\mathcal{N}(0,I)$, so gradients can reach the encoder.
- Decode the code into a reconstruction or the parameters of an observation distribution.
- Minimize reconstruction error for an autoencoder; for a VAE, minimize expected negative log likelihood plus a KL penalty toward the prior.
$$\mathcal{L}_{\mathrm{VAE}}=-\mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]+D_{\mathrm{KL}}(q_\phi(z\mid x)\Vert p(z))$$
The knobs that matter
- latent dimension
- sets code capacity and the amount of compression
- reconstruction likelihood
- defines which reconstruction errors the model penalizes
- KL weight
- trades reconstruction against prior matching in a weighted VAE objective
- encoder and decoder capacity
- controls how much structure each side can represent
What interviewers ask
- How does a VAE differ from a regular autoencoder?
- Why do we use the reparameterization trick?
- What is the ELBO, and how does it relate to log likelihood?
- What is posterior collapse, and how would you detect it?
Common mistakes
- A plain autoencoder does not ensure that arbitrary latent samples decode into useful outputs.
- Mixing sums and means across dimensions can unintentionally change the balance of reconstruction and KL terms.
- A very small KL term can mean the decoder is ignoring the latent code rather than learning a useful representation.
When to use it
Use autoencoders for learned representations, reconstruction or denoising, and VAEs when you need a probabilistic latent model. Reconstruction quality alone does not establish that latent features are useful or that anomaly scores are reliable.
Implement it (7 problems)
- EMA Codebook Updates in VQ-VAEProMediumSolve