Provided by

arXiv

arxiv.org
𝕏gh
0.0/5
0 views
0 saved
Freeadvanced
ResearchersAI EngineersData ScientistsStudents
Introduces Adam, an adaptive learning rate optimization algorithm for efficient stochastic gradient-based training of deep neural networks.

Overview

Adam (Adaptive Moment Estimation) is a groundbreaking optimization algorithm for training deep learning models, leveraging adaptive estimates of lower-order moments. It's celebrated for its computational efficiency, minimal memory requirements, and robustness to noisy or sparse gradients. Adam has become a de facto standard in deep learning, enabling faster convergence and improved performance across a wide range of tasks and model architectures. Its intuitive hyperparameters and strong empirical performance have made it indispensable for AI engineers and researchers alike.

Abstract

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyperparameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms are discussed. We present empirical results for a range of convex optimization problems and for training deep neural networks.