Project · 27 steps · Hard

Direct Preference Optimization (DPO) from Scratch

Implement Direct Preference Optimization end-to-end: log-prob utilities, a policy model, Bradley–Terry preferences, the DPO loss and gradients, IPO variants, and a full train/eval pipeline. Build the math that aligns language models to human preferences without a separate reward model or RL loop.

Each step is one function with tests. Your code runs in the browser, and every step builds on the ones before it.

Part 1: Log-Probability Building Blocks

Numerically stable softmax/log-softmax, token log-prob gathering, and masked sequence log-probabilities used everywhere downstream.

  1. log_softmax
  2. softmax
  3. gather_token_logprobs
  4. masked_sequence_logprob

Part 2: Policy Model & Sequence Log-Probs

Initialize policy parameters, compute token logits, sequence log-probs, and their gradients w.r.t. the policy.

  1. init_policy_params
  2. policy_token_logits
  3. policy_sequence_logprob
  4. sequence_logprob_grad

Part 3: Preference Data & Bradley–Terry

Bradley–Terry pairwise loss, reward accuracy, and utilities to build and sample preference pairs.

  1. bradley_terry_loss
  2. reward_accuracy
  3. build_preference_pairs
  4. sample_preference_batch

Part 4: DPO Loss Formulation

Frozen reference log-probs, policy–reference log-ratios, pair margins, the DPO objective, and its gradient.

  1. freeze_reference_logprobs
  2. policy_reference_logratio
  3. dpo_pair_margin
  4. dpo_loss
  5. dpo_loss_grad

Part 5: DPO Training Loop

A single DPO train step and the outer training loop over preference batches.

  1. dpo_train_step
  2. train_dpo

Part 6: Variants, Implicit Rewards & Diagnostics

Length-normalized log-probs, IPO loss, implicit rewards, preference accuracy, KL to reference, and reward-margin statistics.

  1. length_normalized_logprob
  2. ipo_loss
  3. implicit_reward
  4. preference_accuracy
  5. kl_to_reference
  6. reward_margin_stats

Part 7: Evaluation & Full Pipeline

Aggregate DPO evaluation metrics and wire everything into a complete train-and-evaluate pipeline.

  1. evaluate_dpo
  2. run_dpo_pipeline