Paper
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby
arXiv:2010.11929 · 0▲ · cs.CV, cs.AI, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This paper introduces the Vision Transformer (ViT), which applies a standard Transformer encoder directly to image classification by splitting an image into fixed-size patches (for example 16x16 pixels), linearly embedding each patch as a token, adding position embeddings, and processing the sequence with self-attention. It deliberately avoids convolutions and image-specific inductive biases except for the initial patch extraction and position embedding interpolation at fine-tuning time.
Why it matters
It shows that convolutions are not required for competitive image recognition, letting practitioners reuse the same scalable Transformer architecture and infrastructure across NLP and vision. The catch is that this only works with large-scale pretraining: ViT underperforms comparable ResNets when trained on ImageNet-sized data alone, so the benefit accrues to those who can pretrain on 14M to 300M image datasets or download such pretrained weights.
Practical takeaway
You can fine-tune the released ViT models (pretrained on ImageNet-21k or JFT-300M) for downstream classification tasks instead of a ResNet, and expect strong transfer at lower pretraining compute. Watch for the data-size threshold: without a large pretraining corpus, a ResNet with convolutional inductive bias will likely beat ViT.
Key result
ViT-H/14 pretrained on the (private) JFT-300M dataset reaches 88.55% top-1 on ImageNet versus 87.54% for BiT-L (ResNet152x4) and roughly 88.4 to 88.5% for Noisy Student EfficientNet-L2, while using about 2.5k TPUv3-core-days versus 9.9k to 12.3k for the baselines. Caveat: the flagship result depends on the non-public JFT-300M dataset, though the ImageNet-21k pretrained ViT-L/16 still performs well at much lower cost.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.