Paper
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou
arXiv:2201.11903 · 0▲ · cs.CL, cs.AI
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
The paper introduces chain-of-thought prompting: instead of giving a large language model few-shot examples that map questions directly to answers, each exemplar includes the intermediate reasoning steps that lead to the answer. The model then generates its own step-by-step reasoning before producing a final answer, with no finetuning involved.
Why it matters
This unlocks multi-step reasoning (arithmetic, commonsense, symbolic) from off-the-shelf models using only a handful of hand-written prompt examples, avoiding the cost of building large annotated training sets or maintaining task-specific finetuned checkpoints. The catch for practitioners is scale: the benefit only appears in very large models (roughly 100B parameters and up) and actually hurts smaller models.
Practical takeaway
You can add a few worked-out reasoning examples to your prompt to improve reasoning tasks, and it is robust to who writes the examples and their exact wording, so heavy prompt engineering is not required. Expect this to only pay off on large models and on genuinely multi-step problems; for one or two-step tasks or small models the gain is negligible or negative. Adding an external calculator to evaluate the generated equations boosts arithmetic accuracy further.
Key result
On GSM8K math word problems, PaLM 540B with 8 chain-of-thought exemplars reached 56.9% solve rate versus 17.9% with standard prompting, beating the prior best of a finetuned GPT-3 with a verifier (55%). Caveat: generated reasoning is not guaranteed correct (analysis of wrong answers found 54% had major semantic or coherence errors), and gains did not transfer cleanly across all model families or to CSQA.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.