Gradient Descent with Momentum

Gradient Descent with Momentum | CodeSignal Learn

Lesson Introduction

Welcome back! Today, we'll explore Gradient Descent with Momentum . You've already familiarized yourself with basic gradient descent. However, sometimes gradient descent is slow and gets stuck, especially on bumpy paths to the minimum. So, how do we speed it up? We use momentum . Imagine pushing a heavy shopping cart. Instead of stopping and starting, you build momentum. This helps you move faster. By the end of this lesson, you'll understand how gradient descent with momentum works, implement it in Python, and see how it improves optimization.

Introducing Momentum

v_t = \beta v_{t-1} - \alpha \nabla f(\theta_{t-1})

How Velocity Works in Gradient Descent with Momentum

v_t

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Momentum in optimization is like a push in physical movement. It reduces oscillations (back-and-forth movements) as you head toward the minimum point. Essentially, it helps you move down faster and more steadily.

Let's look at the formulas for momentum:

v_t = \beta v_{t-1} - \alpha \nabla f(\theta_{t-1})

\theta_t = \theta_{t-1} + v_t

Here's what these terms mean:

$v_t$ : Velocity at time step $t$ . It shows speed and direction.
$\theta$ : This is the common symbol for the parameters of the function we're optimizing, similar to variables like $x$ and $y$ in a function $f(x, y)$ .
$\beta$ : Momentum term, usually between 0 and 1. Higher values mean more momentum.
$\alpha$ : Learning rate, controlling step size.
$\nabla f(\theta_{t-1})$ : Gradient of function $f$ at the current point.
$\theta_t$ : Updated parameter at time step $t$ , bringing us closer to the minimum.

Python Implementation: part 1

Now, let's implement Gradient Descent with Momentum in Python. Here's the code snippet: Python # Gradient descent with momentum def gradient_descent_with_momentum(f_grad, init_point, learning_rate=0.1, momentum=0.9, iterations=100): point = list(init_point) velocity = [0] * len(point) for _ in range(iterations): grad = f_grad(point) for i in range(len(point)): velocity[i] = momentum * velocity[i] - learning_rate * grad[i] point[i] += velocity[i] return point # Gradient descent with momentum def gradient_descent_with_momentum(f_grad, init_point, learning_rate=0.1, momentum=0.9, iterations=100): point = list(init_point) velocity = [0] * len(point) for _ in range(iterations): grad = f_grad(point) for i in range(len(point)): velocity[i] = momentum * velocity[i] - learning_rate * grad[i] point[i] += velocity[i] return point Here’s a breakdown of the key lines in the code: velocity = [0] * len(point) : Initializes the velocity vector with zeros, having the same length as the starting point. velocity[i] = momentum * velocity[i] - learning_rate * grad[i] : Updates the velocity by applying the momentum and subtracting the gradient scaled by the learning rate. point[i] += velocity[i] : Updates the current point using the newly calculated velocity.

Python Implementation: part 2

Here's the continuation of our implementation with the example function and initial point: Python # Example function: f(x, y) = x^2 + y^2 def quadratic_gradient(point): x, y = point return [2 * x, 2 * y] # Initial point init_point = [2, 2] # Find the optimal point using gradient descent with momentum optimal_point_momentum = gradient_descent_with_momentum(quadratic_gradient, init_point, learning_rate=0.1, iterations=100) print("Optimal point after gradient descent with momentum:", optimal_point_momentum) # Optimal point after gradient descent with momentum: [~0, ~0] # Example function: f(x, y) = x^2 + y^2 def quadratic_gradient(point): x, y = point return [2 * x, 2 * y] # Initial point init_point = [2, 2] # Find the optimal point using gradient descent with momentum optimal_point_momentum = gradient_descent_with_momentum(quadratic_gradient, init_point, learning_rate=0.1, iterations=100) print("Optimal point after gradient descent with momentum:", optimal_point_momentum) # Optimal point after gradient descent with momentum: [~0, ~0]

To visually understand this, let's compare the paths of basic gradient descent and gradient descent with momentum. We will take $f(x, y) = x^2 + y^2$ and apply both types of gradient with just 3 iterations:

You can clearly see how the gradient descent with momentum converges faster.

Lesson Introduction

Introducing Momentum

How Velocity Works in Gradient Descent with Momentum

Join the 1M+ learners on CodeSignal

Python Implementation: part 1

Python Implementation: part 2

Benefits of Using Momentum

Visualizing Momentum

Lesson Summary