Paper
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei
arXiv:2005.14165 · 0▲ · cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This is the GPT-3 paper: the authors train an autoregressive transformer language model with 175 billion parameters and evaluate it on over two dozen NLP tasks using in-context learning, where the model gets task demonstrations in its prompt at inference time with no gradient updates. They test three settings (zero-shot, one-shot, and few-shot with 10 to 100 examples in the context window) across eight model sizes to measure how performance scales.
Why it matters
The practical shift is that a single frozen model can be pointed at many tasks by writing prompts instead of collecting labeled datasets and running fine-tuning jobs per task. This trades the engineering cost of per-task training pipelines for prompt design, though the model itself is expensive and inconvenient to run inference on at this scale.
Practical takeaway
You can prototype a task by writing a natural language instruction plus a few examples in the prompt rather than assembling a training set, but expect weak spots: comparison-style tasks (whether one sentence implies another, whether a word means the same in two sentences) stay near chance, and results have high variance.
Key result
On TriviaQA (closed-book, no fine-tuning), few-shot GPT-3 reached 71.2% accuracy versus 68.0% for a fine-tuned open-domain retrieval system (RAG) and 60.5% for fine-tuned T5-11B. Caveat: on many other benchmarks (SuperGLUE, RACE, ANLI, WiC) few-shot GPT-3 lags fine-tuned models, sometimes badly, so the headline competitiveness is task-dependent.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.