Advanced Optimization: Understanding and Implementing ADAM

Introduction to ADAM

Hello! Today, we will explore the ADAM (Adaptive Moment Estimation) algorithm. This advanced optimization algorithm is a favorite among machine learning practitioners as it combines the advantages of two other extensions of Stochastic Gradient Descent (SGD): Root Mean Square Propagation (RMSprop) and Adaptive Gradient Algorithm (AdaGrad). Our primary focus today is understanding ADAM, and we will also build it from scratch in C++ to optimize multivariable functions.

Understanding ADAM

Before we dive into ADAM, let us recall that classic gradient descent methods like SGD and even sophisticated versions like Momentum and RMSProp have some limitations. These limitations relate to sensitivity to learning rates, the issue of vanishing gradients, and the absence of individual adaptive learning rates for different parameters.

ADAM, a promising choice for an optimization algorithm, combines the merits of RMSProp and AdaGrad. It maintains a per-parameter learning rate adapted based on the average of recent magnitudes of the gradients for the weights (similar to RMSProp) and the average of recent gradients (like Momentum). This mechanism enables the algorithm to traverse quickly over the low gradient regions and slow down near the optimal points.

ADAM Mathematically

For ADAM, we modify the update rule of SGD, introducing two additional hyperparameters, beta1 and beta2. The hyperparameter beta1 controls the exponential decay rate for the first-moment estimates (similar to Momentum), while beta2 controls the exponential decay rate for the second-moment estimates (similar to RMSProp). The standard ADAM algorithm always includes bias correction for these moment estimates, which is crucial for proper convergence, especially in the early stages of training.

The mathematical expression, including bias correction, is as follows:

mt=β1∗mt−1+(1−β1)∗gradm_t = \beta_1 * m_{t-1} + (1 - \beta_1) * grad vt=β2∗vt−1+(1−β2)∗grad2v_t = \beta_2 * v_{t-1} + (1 - \beta_2) * grad^2 m^t=mt1−β1t\hat{m}_t = \frac{m_t}{1 - \beta_1^t} v^t=vt1−β2t\hat{v}_t = \frac{v_t}{1 - \beta_2^t} w=w−α∗m^tv^t+ϵw = w - \alpha * \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

Here, m_t and v_t are estimates of the gradients' first moment (the mean) and the second moment (the uncentered variance), respectively, while grad represents the gradient. The terms m^t\hat{m}_t and v^t\hat{v}_t are the bias-corrected estimates. We also use an epsilon constant to maintain numerical stability and prevent division by zero, as in RMSProp.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal