Noise Prediction Loss for Diffusion Training

MediumML Coding

Problem statement

Implement a function that computes the noise prediction training loss used in denoising diffusion probabilistic models (DDPM).

During training, a diffusion model learns to reverse a gradual noising process. Given a clean data sample, a specific timestep, and random Gaussian noise, the model must predict the noise that was added.

Your function noise_prediction_loss(x_0, alpha_bar, t, epsilon, epsilon_pred) should:

  1. Look up the cumulative noise schedule value alpha_bar_t for each sample's timestep
  2. Construct the noisy version of each sample x_t using the closed-form reparameterization from the forward diffusion process
  3. Compute the mean squared error (MSE) between the actual noise and the predicted noise, averaged over all elements in the batch
  4. Return a tuple of (x_t, loss) where x_t is a numpy array of the noisy samples and loss is a scalar float

Inputs:

  • x_0: numpy array of shape (B, D) containing B clean data samples with D features each
  • alpha_bar: numpy array of shape (T,) containing the cumulative product noise schedule for all T timesteps
  • t: numpy array of shape (B,) containing integer timestep indices for each sample in the batch
  • epsilon: numpy array of shape (B, D) containing the actual Gaussian noise added to each sample
  • epsilon_pred: numpy array of shape (B, D) containing the model's predicted noise for each sample

The noisy sample at timestep t is formed by mixing the original signal and noise according to the schedule. The training loss measures how well the model's noise prediction matches the true noise.

Example

Input:

x_0 = np.array([[1.0, 0.0]]), alpha_bar = np.array([0.99, 0.95, 0.80, 0.50, 0.10]), t = np.array([3]), epsilon = np.array([[0.5, -0.5]]), epsilon_pred = np.array([[0.4, -0.6]])

Output:

(array([[ 1.0607, -0.3536]]), 0.01)

Reasoning: At timestep t=3, alpha_bar_t = 0.50. The noisy sample is computed as x_t = sqrt(0.5) [1.0, 0.0] + sqrt(0.5) [0.5, -0.5] = [0.7071 + 0.3536, 0.0 - 0.3536] = [1.0607, -0.3536]. The MSE loss between the true noise [0.5, -0.5] and predicted noise [0.4, -0.6] is mean((0.1)^2 + (0.1)^2) = 0.01.

Show hint

What closed-form expression lets you produce a noisy sample at an arbitrary timestep without simulating every earlier step?

Relevant roles: ML Engineer, Deep Learning Engineer, Gen AI Engineer and Research Scientist.

Related free problems