Paper
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela
arXiv:2005.11401 · 0▲ · cs.CL, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This paper introduces Retrieval-Augmented Generation (RAG), a fine-tuning recipe that pairs a pre-trained seq2seq generator (BART-large) with a neural retriever (Dense Passage Retriever) that pulls passages from a dense vector index of Wikipedia. It presents two variants: RAG-Sequence, which conditions the whole output on one retrieved document, and RAG-Token, which can draw a different document per generated token, with both retriever and generator trained end-to-end using only input/output pairs.
Why it matters
For a practitioner, this is the architecture that lets a generation model cite external, swappable knowledge instead of baking all facts into weights. You can update what the model knows by replacing the document index at test time without retraining, and the retrieved text is human-readable so you can inspect why an answer was produced. It also hits strong QA accuracy with far fewer trainable parameters than giant closed-book models.
Practical takeaway
You can now add a document index to a generator and get factual grounding plus updatable knowledge without training on retrieval labels; the implementation is open-sourced in HuggingFace Transformers. Watch for retrieval collapse on tasks with weak factual signal (long story generation), where the retriever degenerates to returning the same documents and the model falls back to BART-level behavior.
Key result
RAG-Sequence reached 44.5 Exact Match on open Natural Questions, beating REALM (40.4) and DPR (41.5), using 626M trainable parameters versus T5-11B's 11B (closed-book T5-11B scored 34.5). Note the retriever was initialized from DPR, which itself was trained with retrieval supervision on NQ and TriviaQA, so RAG is not fully supervision-free at initialization.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.