Paper
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever
arXiv:2103.00020 · 0▲ · cs.CV, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This is the CLIP paper. It trains an image encoder and a text encoder jointly on 400 million (image, text) pairs scraped from the internet, using a contrastive objective that predicts which caption goes with which image within a batch rather than predicting exact caption words. After pre-training, you build a classifier for any dataset by feeding the class names (as text prompts) through the text encoder, so the model classifies images it was never explicitly trained to label.
Why it matters
You can build an image classifier for a new task without collecting or labeling training data: just write out the class names in natural language. This removes the retrain-per-task step for many classification and retrieval workflows, though it comes with real limits on fine-grained and abstract tasks.
Practical takeaway
You can now do zero-shot image classification and image/text retrieval by supplying candidate labels as text, and prompt wording matters (adding context like 'a photo of a {label}, a type of pet' and ensembling prompts added about 5 points on ImageNet). Watch for weak performance on specialized tasks (satellite imagery, counting, MNIST digits) where it can drop to near-random.
Key result
Zero-shot CLIP matches the original ResNet-50's ImageNet top-1 accuracy (76.2%) while using none of the 1.28M labeled training images. Caveat: this parity holds on ImageNet-style object classification; on 27 diverse datasets zero-shot still trails fully supervised linear probes by 10 to 25 points on most, and the model authors admit ~1000x more compute would be needed to reach overall SOTA.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.