Paper
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, Quoc V. Le
arXiv:1409.3215 · 0▲ · cs.CL, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This paper introduces the sequence-to-sequence (seq2seq) encoder-decoder architecture: one multilayer LSTM reads an input sentence into a fixed-length vector, and a second LSTM decodes that vector into the output sentence one word at a time. It applies this to English-to-French machine translation and adds a key trick of reversing the word order of the source sentence before feeding it in.
Why it matters
This showed that a single general-purpose neural network could handle variable-length input and output without task-specific engineering, replacing the hand-built pipelines of phrase-based statistical machine translation. The encoder-decoder pattern here became the template that later attention and transformer models built on, so it is foundational rather than something you would deploy today.
Practical takeaway
Understand this as the direct ancestor of every encoder-decoder model you use now; the source-reversal trick is a reminder that reducing the distance between related tokens (minimizing time lag) can matter as much as raw model capacity, though attention later made the trick obsolete.
Key result
An ensemble of 5 reversed LSTMs with beam search scored 34.81 BLEU on WMT'14 English-to-French, beating the phrase-based SMT baseline of 33.30, and reached 36.5 when used to rescore the SMT system's 1000-best lists (versus the best published 37.0). The direct-translation score was penalized on out-of-vocabulary words since the model used only an 80k target vocabulary.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.