Paper
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe, Christian Szegedy
arXiv:1502.03167 · 0▲ · cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This paper introduces Batch Normalization, a layer inserted before each nonlinearity that normalizes activations to zero mean and unit variance using per-mini-batch statistics, then applies a learned scale and shift. The normalization is part of the network architecture so gradients backpropagate through it, and at inference time it uses population statistics computed from moving averages.
Why it matters
BatchNorm lets you train deep networks with much higher learning rates and far less careful weight initialization, cutting training time substantially. It also acts as a regularizer, often letting you remove or reduce Dropout, and makes saturating nonlinearities like sigmoid trainable in deep nets where they previously failed.
Practical takeaway
Add a BN layer before each nonlinearity in your convolutional or fully connected network, then raise the learning rate and drop Dropout and Local Response Normalization. Watch that inference uses frozen population statistics rather than mini-batch statistics so outputs are deterministic.
Key result
On ImageNet classification, an Inception variant with BatchNorm plus a 5x higher learning rate (BN-x5) reached Inception's 72.2% accuracy in 14x fewer training steps; a 30x learning rate version hit 74.8%. An ensemble of 6 BatchNorm networks reached 4.9% top-5 validation error (4.82% test), beating the prior best of 4.94%.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.