Paper
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe
arXiv:2203.02155 · 0▲ · cs.CL, cs.AI, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This is the InstructGPT paper, which fine-tunes GPT-3 to follow instructions using reinforcement learning from human feedback (RLHF). The pipeline has three stages: supervised fine-tuning on labeler-written demonstrations, training a reward model on human rankings of model outputs, and then optimizing the policy against that reward model using PPO.
Why it matters
It shows that alignment fine-tuning is far cheaper and more effective than scaling: a 1.3B model tuned this way beats a 175B base model on human preference, so you get better instruction-following without paying for a 100x larger model at inference. This is the direct precursor to the ChatGPT-style assistants that now dominate practitioner workflows, and it established RLHF as the standard recipe for turning a raw language model into a usable product.
Practical takeaway
Watch for the alignment tax: raw RLHF caused regressions on public NLP benchmarks (SQuAD, DROP, translation), which the authors only mitigated by mixing pretraining gradients back into the PPO updates (their PPO-ptx variant). If you tune your own models with preference data, expect to blend in pretraining data to avoid capability loss, and note that these models still follow harmful instructions because helpfulness was prioritized during training.
Key result
175B InstructGPT outputs were preferred to 175B GPT-3 outputs 85 +/- 3% of the time, and to few-shot-prompted GPT-3 71 +/- 4% of the time, judged by the same 40 labelers who generated the training data on OpenAI's own API prompt distribution (so preferences reflect that specific labeler group, not a broad human population). Truthfulness roughly doubled on TruthfulQA and closed-domain hallucination dropped from 41% to 21%, but bias on Winogender and CrowS-Pairs did not improve.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.