Basic Gradient Descent

Basic Gradient Descent | CodeSignal Learn

Lesson Introduction

Welcome to the lesson on Basic Gradient Descent! In this lesson, we will learn about an optimization algorithm called gradient descent . It helps us find the minimum value of a function, which is very important in fields such as machine learning and data science. Imagine you're on a hike in the mountains. You want to find the lowest point in a valley. Gradient descent is like taking small steps downhill until you reach the bottom. By the end of this lesson, you'll understand what gradient descent is, how it works, and how to implement it in Python . Let's get started!

Understanding Gradient Descent

f(x, y)

Example: Simple Quadratic Function

f(x, y) = x^2 + y^2

Gradient Calculation

f(x, y) = x^2 + y^2

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

To apply gradient descent, we need to calculate the gradient of the function. For the quadratic function $f(x, y) = x^2 + y^2$ , the gradient is:

$\nabla f(x, y) = [2x, 2y]$

The gradient tells us how the function changes at a given point.

Imagine rolling a ball down a hill. The direction in which the ball accelerates the fastest is the gradient direction. The steeper the slope, the larger the gradient.

Implementing Gradient Descent: Part 1

\mathbf{x}_{t+1} = \mathbf{x}_t - \eta \nabla f(\mathbf{x}_t)

Implementing Gradient Descent: Part 2

Next, we will define the function we want to minimize and its gradient. Python def quadratic_function(point): x, y = point return x**2 + y**2 def quadratic_gradient(point): x, y = point return [2 * x, 2 * y] def quadratic_function(point): x, y = point return x**2 + y**2 def quadratic_gradient(point): x, y = point return [2 * x, 2 * y] Now, we run the gradient descent algorithm. Python optimal_point = simple_gradient_descent(quadratic_gradient, [2, 2], learning_rate=0.1, iterations=100) print("Optimal point after basic gradient descent:", optimal_point) # Optimal point after basic gradient descent: close to [0, 0] optimal_point = simple_gradient_descent(quadratic_gradient, [2, 2], learning_rate=0.1, iterations=100) print("Optimal point after basic gradient descent:", optimal_point) # Optimal point after basic gradient descent: close to [0, 0]

Now, let's see how to implement a basic version of gradient descent in Python. Let's see the formula first:

$\mathbf{x}_{t+1} = \mathbf{x}_t - \eta \nabla f(\mathbf{x}_t)$

where:

$\mathbf{x}_t$ is the current point.
$\eta$ is the learning rate.
$\nabla f(\mathbf{x}_t)$ is the gradient at the current point.

We'll start by initializing a point and updating it by moving against the gradient.

Python

# Gradient descent without path output
def simple_gradient_descent(f_grad, init_point, learning_rate=0.1, iterations=100):
    point = list(init_point)
    for _ in range(iterations):
        grad = f_grad(point)
        point = [p - learning_rate * g for p, g in zip(point, grad)]
    return point

Here are the inputs of the simple_gradient_descent function:

f_grad: The gradient function of the target function.
init_point: The starting point for the gradient descent algorithm.
learning_rate: A hyperparameter that controls the step size in each iteration.
iterations: The number of iterations to update the point.

In simple_gradient_descent:

We take in the gradient function, initial point, learning rate, and number of iterations.
We initialize the point and update it using the gradient and learning rate.
After iterating, we return the optimal point.

For the example functions:

quadratic_function computes $x^2 + y^2$ .
quadratic_gradient calculates the gradient of this function.

Finally, let's visualize the gradient descent path. We will mark each gradient step as a point on the plot, starting with $[2, 2]$ . To make the plot clear, let's limit the gradient descent to 10 iterations.

You can observe how the gradient moves towards the minimum point. As we approach the minimum point, the function's steepness decreases, so we take smaller steps. If we take more iterations, we will find this minimum point efficiently, as demonstrated earlier.

Lesson Introduction

Understanding Gradient Descent

Example: Simple Quadratic Function

Gradient Calculation

Join the 1M+ learners on CodeSignal

Plotting the Target Function

Implementing Gradient Descent: Part 1

Choosing the Starting Point and Learning Rate

Implementing Gradient Descent: Part 2

Code Explanation

Visualizing the Gradient Descent Path

Lesson Summary