Paper
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
arXiv:1810.04805 · 0▲ · cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
BERT is a language representation model that pre-trains a deep bidirectional Transformer encoder using two self-supervised tasks: masked language modeling (predicting randomly hidden tokens from both left and right context) and next sentence prediction. The pre-trained model is then fine-tuned end-to-end for specific tasks by adding a single output layer, with no task-specific architecture changes.
Why it matters
It replaces the pattern of building custom architectures per NLP task with a workflow of pre-train once, then fine-tune the whole model cheaply for each downstream task. Fine-tuning takes roughly an hour on a single Cloud TPU or a few hours on a GPU, so practitioners can reach competitive results on classification, tagging, and question answering without heavy engineering or large labeled datasets.
Practical takeaway
You can start from a released pre-trained checkpoint and fine-tune with one added output layer for classification, span prediction, or sequence tagging, rather than designing a bespoke model. Note that bigger models helped even on small datasets (3,600 examples), so favor the larger checkpoint when your labeled data is scarce.
Key result
BERT-LARGE pushed the GLUE benchmark score to 80.5 (up 7.7 points over the prior state of the art) and MNLI accuracy to 86.7 (up 4.6 points); ablations show most of the gain comes from bidirectionality and the two pre-training tasks rather than from the larger corpus or batch size versus GPT.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.