Noise Prediction Loss for Diffusion Training
Problem statement
Implement a function that computes the noise prediction training loss used in denoising diffusion probabilistic models (DDPM).
During training, a diffusion model learns to reverse a gradual noising process. Given a clean data sample, a specific timestep, and random Gaussian noise, the model must predict the noise that was added.
Your function noise_prediction_loss(x_0, alpha_bar, t, epsilon, epsilon_pred) should:
- Look up the cumulative noise schedule value alpha_bar_t for each sample's timestep
- Construct the noisy version of each sample x_t using the closed-form reparameterization from the forward diffusion process
- Compute the mean squared error (MSE) between the actual noise and the predicted noise, averaged over all elements in the batch
- Return a tuple of (x_t, loss) where x_t is a numpy array of the noisy samples and loss is a scalar float
Inputs:
- x_0: numpy array of shape (B, D) containing B clean data samples with D features each
- alpha_bar: numpy array of shape (T,) containing the cumulative product noise schedule for all T timesteps
- t: numpy array of shape (B,) containing integer timestep indices for each sample in the batch
- epsilon: numpy array of shape (B, D) containing the actual Gaussian noise added to each sample
- epsilon_pred: numpy array of shape (B, D) containing the model's predicted noise for each sample
The noisy sample at timestep t is formed by mixing the original signal and noise according to the schedule. The training loss measures how well the model's noise prediction matches the true noise.
Example
Input:
x_0 = np.array([[1.0, 0.0]]), alpha_bar = np.array([0.99, 0.95, 0.80, 0.50, 0.10]), t = np.array([3]), epsilon = np.array([[0.5, -0.5]]), epsilon_pred = np.array([[0.4, -0.6]])
Output:
(array([[ 1.0607, -0.3536]]), 0.01)
Reasoning: At timestep t=3, alpha_bar_t = 0.50. The noisy sample is computed as x_t = sqrt(0.5) [1.0, 0.0] + sqrt(0.5) [0.5, -0.5] = [0.7071 + 0.3536, 0.0 - 0.3536] = [1.0607, -0.3536]. The MSE loss between the true noise [0.5, -0.5] and predicted noise [0.4, -0.6] is mean((0.1)^2 + (0.1)^2) = 0.01.
Show hint
What closed-form expression lets you produce a noisy sample at an arbitrary timestep without simulating every earlier step?
Relevant roles: ML Engineer, Deep Learning Engineer, Gen AI Engineer and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium