Hello! Today, we will explore the ADAM (Adaptive Moment Estimation) algorithm. This advanced optimization algorithm is a favorite among machine learning practitioners as it combines the advantages of two other extensions of Stochastic Gradient Descent (SGD): Root Mean Square Propagation (RMSprop) and Adaptive Gradient Algorithm (AdaGrad). Our primary focus today is understanding ADAM, and we will also build it from scratch in Python to optimize multivariable functions.

Before we dive into ADAM, let us recall that classic gradient descent methods like SGD and even sophisticated versions like Momentum and RMSProp have some limitations. These limitations relate to sensitivity to learning rates, the issue of vanishing gradients, and the absence of individual adaptive learning rates for different parameters.

ADAM, a promising choice for an optimization algorithm, combines the merits of RMSProp and AdaGrad. It maintains a per-parameter learning rate adapted based on the average of recent magnitudes of the gradients for the weights (similar to RMSProp) and the average of recent gradients (like Momentum). This mechanism enables the algorithm to traverse quickly over the low gradient regions and slow down near the optimal points.

For ADAM, we modify the update rule of SGD, introducing two additional hyperparameters, beta1 and . The hyperparameter controls the exponential decay rate for the first-moment estimates (similar to Momentum), while controls the exponential decay rate for the second-moment estimates (similar to RMSProp). The mathematical expression can be formulated as follows: