Paper
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn
arXiv:2305.18290 · 0▲ · cs.LG, cs.AI, cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This paper introduces Direct Preference Optimization (DPO), a method for aligning language models to human preferences using a single binary cross-entropy classification loss instead of the standard RLHF pipeline. It shows that the KL-constrained reward maximization objective used in RLHF has a closed-form optimal policy, which lets you reparameterize the reward as a function of the policy itself and skip both the separate reward model and the reinforcement learning loop.
Why it matters
RLHF with PPO requires training a separate reward model, sampling from the model during training, and careful hyperparameter tuning, all of which are expensive and unstable. DPO replaces this with a supervised-style loss that runs on a fixed offline preference dataset, so alignment becomes a straightforward fine-tuning job with no online sampling and minimal tuning.
Practical takeaway
You can align a model directly on preference pairs (chosen vs rejected) with a loss that only needs log-probabilities from the policy and a frozen reference model, no reward model or PPO rollout. DPO is now standard in libraries like Hugging Face TRL, so it is a practical default when you have a preference dataset.
Key result
On Reddit TL;DR summarization, DPO reached roughly 61% win rate against human reference summaries (GPT-4 judged) at temperature 0, versus PPO's 57% at its best temperature, and was more robust to sampling temperature. Evaluations used models up to 6B parameters and relied on GPT-4 as an evaluator proxy (validated against a small human study), so results are on relatively modest models and preference-quality benchmarks rather than large frontier systems.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.