Paper
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
arXiv:1706.03762 · 0▲ · cs.CL, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This paper introduces the Transformer, a sequence-to-sequence architecture for machine translation that replaces recurrent and convolutional layers entirely with stacked self-attention and position-wise feed-forward layers. It uses scaled dot-product attention with multiple attention heads, plus sinusoidal positional encodings to inject word order since there is no recurrence.
Why it matters
Because self-attention connects all positions in a constant number of sequential operations rather than one step per token, training parallelizes across a sequence instead of running strictly left to right. For a practitioner this means much shorter training times on the same hardware and better handling of long-range dependencies, which is why this architecture became the foundation for essentially every large language model that followed.
Practical takeaway
This is the base design behind modern LLMs, so understanding queries/keys/values, multi-head attention, and positional encodings is the prerequisite for reading almost any current model paper. Note the O(n^2) cost in sequence length, which is the bottleneck later work (sparse attention, FlashAttention) exists to address.
Key result
28.4 BLEU on WMT 2014 English-to-German, beating the previous best (including ensembles) by over 2 BLEU, and 41.8 BLEU on English-to-French as a single model, trained in 3.5 days on 8 P100 GPUs at a fraction of the FLOPs of prior state-of-the-art systems. Results are limited to two translation tasks plus one constituency parsing test, not the broad general-purpose evaluation the architecture later became known for.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.