Paper
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba
arXiv:1412.6980 · 0▲ · cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This paper introduces Adam, a first-order stochastic optimization algorithm that maintains per-parameter adaptive learning rates using exponential moving averages of the gradient (first moment) and squared gradient (second raw moment), with a bias-correction step for the zero-initialized averages. It combines ideas from AdaGrad (sparse gradients) and RMSProp (non-stationary objectives), and also presents AdaMax, a variant based on the infinity norm.
Why it matters
Adam gives you a gradient descent optimizer that needs little per-parameter tuning, uses little memory, and works out of the box across many model types, which is why it became the default optimizer for training neural networks. The effective step size is roughly bounded by the stepsize hyperparameter, so choosing a learning rate is more predictable than with plain SGD.
Practical takeaway
Use the published defaults (alpha=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8) as a starting point for almost any deep learning training run. Keep bias correction on, especially when beta2 is close to 1 for sparse gradients, since dropping it (as RMSProp does) can cause very large early steps and divergence.
Key result
On MNIST multilayer neural networks with dropout, Adam converged faster than AdaGrad, RMSProp, SGD with Nesterov momentum, and AdaDelta; against the SFO quasi-Newton method it made faster progress while SFO ran 5-10x slower per iteration with memory linear in minibatch count. Caveats: evaluations are on small 2015-era datasets (MNIST, IMDB, CIFAR-10), the O(sqrt(T)) regret bound only covers convex objectives, and on CNNs Adam showed only marginal improvement over SGD with momentum.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.