Understanding and Implementing RMSProp in Python

Introduction to RMSProp

Hello! Today, we will dive into RMSProp (Root Mean Square Propagation). This sophisticated optimization algorithm accelerates convergence by adapting the learning rate for each weight separately, addressing the limitations of previous techniques such as Stochastic Gradient Descent (SGD), Mini-Batch Gradient Descent, and momentum . Our focus today is understanding RMSProp and coding it from scratch in Python to optimize multivariable functions.

Recap on Gradient Descent Techniques

Let's begin with a quick recap: SGD and Mini-Batch Gradient Descent can be sensitive to learning rates and may converge slowly. Even momentum, which mitigates these issues to an extent, has limitations. When a uniform learning rate is applied across all parameters, efficient optimization might not be achieved. This is where RMSProp steps in to offer a solution.

Understanding RMSProp

RMSProp, an advanced optimization algorithm, adjusts the gradient descent step for each weight individually, accelerating training and allowing faster convergence. This optimization is achieved by RMSProp keeping track of a running average of the square of gradients and then using this information to scale the learning rate.

RMSProp Mathematically

s_{dw} = \rho * s_{dw} + (1-\rho){dw}^{2}

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Application of RMSProp on Multivariable Function Optimization

Now let's apply RMSProp to find the minimum of a multivariable function f(x, y) = x^2 + y^2. Corresponding gradients are df/dx = 2*x and df/dy = 2*y. We've set the initial starting point to (x, y) = (5, 4), and picked common choices for hyperparameters (rho = 0.9, epsilon = 1e-6, and learning_rate = 0.1), running our optimizer over 100 epochs.

Python

def f(x, y):
    return x**2 + y**2

def df(x, y):
    return np.array([2*x, 2*y])

coordinates = np.array([5.0, 4.0])
learning_rate = 0.1
rho = 0.9
epsilon = 1e-6
max_epochs = 100

s_prev = np.array([0, 0])

for epoch in range(max_epochs + 1):
    grad = df(coordinates[0], coordinates[1])
    updates, s_prev = RMSProp(learning_rate, rho, epsilon, grad, s_prev)
    coordinates -= updates
    if epoch % 20 == 0:
        print(f"Epoch {epoch}, current state: {coordinates}")

def f(x, y):
    return x**2 + y**2

def df(x, y):
    return np.array([2*x, 2*y])

coordinates = np.array([5.0, 4.0])
learning_rate = 0.1
rho = 0.9
epsilon = 1e-6
max_epochs = 100

s_prev = np.array([0, 0])

for epoch in range(max_epochs + 1):
    grad = df(coordinates[0], coordinates[1])
    updates, s_prev = RMSProp(learning_rate, rho, epsilon, grad, s_prev)
    coordinates -= updates
    if epoch % 20 == 0:
        print(f"Epoch {epoch}, current state: {coordinates}")

The output of this code is as follows:

Epoch 0, current state: [4.68377233 3.68377236]
Epoch 20, current state: [2.3688824  1.47561697]
Epoch 40, current state: [0.95903672 0.35004133]
Epoch 60, current state: [0.13761293 0.00745214]
Epoch 80, current state: [3.91649374e-04 3.12725069e-09]
Epoch 100, current state: [-3.07701828e-17  2.18862195e-20]

Epoch 0, current state: [4.68377233 3.68377236]
Epoch 20, current state: [2.3688824  1.47561697]
Epoch 40, current state: [0.95903672 0.35004133]
Epoch 60, current state: [0.13761293 0.00745214]
Epoch 80, current state: [3.91649374e-04 3.12725069e-09]
Epoch 100, current state: [-3.07701828e-17  2.18862195e-20]

As you can see, x and y quickly approach 0, which is indeed the minimum of the given function.

def RMSProp(learning_rate, rho, epsilon, grad, s_prev): # Update squared gradient s = rho * s_prev + (1 - rho) * np.power(grad, 2) # Calculate updates updates = learning_rate * grad / (np.sqrt(s) + epsilon) return updates, s

Introduction to RMSProp

Recap on Gradient Descent Techniques

Understanding RMSProp

RMSProp Mathematically

Join the 1M+ learners on CodeSignal

RMSProp in Python Code

Application of RMSProp on Multivariable Function Optimization

Evaluation of RMSProp Over Other Gradient Descent Techniques

Conclusion