Optimization

Diederik P. Kingma, Jimmy Ba

This paper introduces Adam, a first-order stochastic optimization algorithm that maintains per-parameter adaptive learning rates using exponential moving averages of the gradient (first moment) and squared gradient (second raw moment), with a bias-correction step for the zero-initialized averages. It combines ideas from AdaGrad (sparse gradients) and RMSProp (non-stationary objectives), and also presents AdaMax, a variant based on the infinity norm.