Chapter 2: Deep learning

From autoencoders to VAEs: the ELBO and reparameterization

You will be able to write the VAE loss and compute it by hand, and explain why training needs the reparameterization trick. A VAE is a model that makes new data from random codes. Interviewers often ask for its loss, for the trick, and for what goes wrong when one part of the loss wins.

7 minInteractive lessonChapter 2: Deep learningLesson 18 of 20

What you will learn

  • Write the VAE loss with a Gaussian KL term
  • Explain why the reparameterization trick allows gradients through sampling
  • Explain posterior collapse

Key terms

code
the short list of numbers the encoder makes from a photo
coordinate
one number of the code, also called a latent dimension
blob
the encoder's guess of where a code is: a centre μ and a spread σ²
prior
the target blob chosen before any photo: centre 0 and variance 1 in each coordinate

Prerequisites

Take Graph neural networks: message passing and link prediction first. It comes right before this lesson in the course.

Where this lesson picks up: You turned graph nodes into learned vectors and scored pairs of them to predict links. Those vectors described things that already exist, so can we instead draw a vector at random and decode it into something new?

What comes next

What if we add noise in many small, planned steps, instead of once, and then learn to remove it?

Next lesson: Diffusion: noise forward, denoise backward

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.