Paper
Improving neural networks by preventing co-adaptation of feature detectors
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov
arXiv:1207.0580 · 0▲ · cs.NE, cs.CV, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This is the original dropout paper: during training, each hidden unit is randomly omitted with probability 0.5 on every training case, which stops units from co-adapting to rely on specific other units. At test time you use the full network with outgoing weights halved, which approximates averaging over the exponentially many thinned networks.
Why it matters
Dropout is a cheap regularizer that lets you train larger networks on limited data without the overfitting that used to require training and averaging many separate models. It removed the need for early stopping in several of the paper's experiments and made results less sensitive to architecture and learning rate choices.
Practical takeaway
You can apply 50% dropout to fully connected hidden layers (and around 20% on inputs) as a default regularizer, but expect little benefit in convolutional layers since weight sharing already limits overfitting. Pair it with a per-unit L2 norm constraint (max squared length) so you can start with a high, decaying learning rate.
Key result
On MNIST without pretraining or data augmentation, dropout cut the best published feedforward result of 160 test errors to about 110. It also set records at the time: 42.4% top error on ImageNet 2010 (vs 45.7% prior state of the art) and 19.7% TIMIT phone recognition error (vs 22.7% without dropout).
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.