Project · 27 steps · Hard
Direct Preference Optimization (DPO) from Scratch
Implement Direct Preference Optimization end-to-end: log-prob utilities, a policy model, Bradley–Terry preferences, the DPO loss and gradients, IPO variants, and a full train/eval pipeline. Build the math that aligns language models to human preferences without a separate reward model or RL loop.
Each step is one function with tests. Your code runs in the browser, and every step builds on the ones before it.
Part 1: Log-Probability Building Blocks
Numerically stable softmax/log-softmax, token log-prob gathering, and masked sequence log-probabilities used everywhere downstream.
log_softmaxsoftmaxgather_token_logprobsmasked_sequence_logprob
Part 2: Policy Model & Sequence Log-Probs
Initialize policy parameters, compute token logits, sequence log-probs, and their gradients w.r.t. the policy.
init_policy_paramspolicy_token_logitspolicy_sequence_logprobsequence_logprob_grad
Part 3: Preference Data & Bradley–Terry
Bradley–Terry pairwise loss, reward accuracy, and utilities to build and sample preference pairs.
bradley_terry_lossreward_accuracybuild_preference_pairssample_preference_batch
Part 4: DPO Loss Formulation
Frozen reference log-probs, policy–reference log-ratios, pair margins, the DPO objective, and its gradient.
freeze_reference_logprobspolicy_reference_logratiodpo_pair_margindpo_lossdpo_loss_grad
Part 5: DPO Training Loop
A single DPO train step and the outer training loop over preference batches.
dpo_train_steptrain_dpo
Part 6: Variants, Implicit Rewards & Diagnostics
Length-normalized log-probs, IPO loss, implicit rewards, preference accuracy, KL to reference, and reward-margin statistics.
length_normalized_logprobipo_lossimplicit_rewardpreference_accuracykl_to_referencereward_margin_stats
Part 7: Evaluation & Full Pipeline
Aggregate DPO evaluation metrics and wire everything into a complete train-and-evaluate pipeline.
evaluate_dporun_dpo_pipeline