Understanding Gradient

Lesson Introduction

Hello and welcome to our lesson on "Understanding Gradient"! In this lesson, we'll explore what gradients are, why they matter, and how to compute them in Python. This will help you understand how to optimize machine learning models. By the end, you'll know how to find the gradient of a multivariable function and understand its importance.

What is a Gradient?

Let's start with the basics. Imagine a landscape of hills and valleys. The gradient tells you the direction of the steepest climb. Formally, the gradient is a vector that contains all the partial derivatives of a multivariable function, pointing in the direction where the function increases the most. If you're climbing a hill and want to get to the top as quickly as possible, the gradient guides you on the steepest path. Gradients are crucial in machine learning. When training a model, you want to find the best parameters that minimize a loss function. The gradient helps you know how to adjust these parameters to reduce error. Think of it as having a map that shows the quickest way to reach a lower point. For example, in adjusting weights in a neural network to improve predictions, the gradient shows how to change the weights.

Example Function and Partial Derivatives as Vectors

f(x, y) = x^2 + y^2

Plotting the Function and Gradient Vector

Let's visualize this concept by plotting the function and gradient vector at the point (1, -1). This plot shows the function as a heatmap and the gradient vector at the point (1, -1) as a white arrow. This arrow points towards the steepest function's ascent, meaning the fastest increase. Conversely, the negative gradient points to the steepest descent, useful for finding the minimum value of the function. When optimizing machine learning models, we often follow the negative gradient direction to minimize errors.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Calculating the Gradient Using Central Difference Formula: part 1

Now, let's see how to calculate the gradient using the central difference method, which takes the difference between values at slightly different points around the point of interest.

Here's how to do this in Python:

Python

# Centrally differenced gradient for multivariable functions
def gradient(f, pt, h=1e-5):
    grad = []
    for i in range(len(pt)):
        augmented_point = pt[:]
        augmented_point[i] += h
        grad.append((f(*augmented_point) - f(*pt)) / h)
    return grad

# Centrally differenced gradient for multivariable functions
def gradient(f, pt, h=1e-5):
    grad = []
    for i in range(len(pt)):
        augmented_point = pt[:]
        augmented_point[i] += h
        grad.append((f(*augmented_point) - f(*pt)) / h)
    return grad

This function computes the gradient of a multivariable function f at a given point pt by approximating partial derivatives. It does so by iterating through each variable, slightly perturbing it, and measuring the change in the function's value. The results are collected in the grad list and returned.

Let's break it down:

grad = []: Initializes an empty list to store the partial derivatives.
for i in range(len(pt)):: Iterates over each variable in the point pt. In our case, it is x and y, but it can be any amount of variables.
augmented_point = pt[:]: Creates a copy of pt to modify without altering the original point.
augmented_point[i] += h: Adds a small value h to the i-th coordinate of the copied point.
grad.append((f(*augmented_point) - f(*pt)) / h): Computes the approximate partial derivative for the i-th coordinate and appends it to the grad list. The star operator is used here to unpack a point into a list of inputs. It turns x, y list into two separate variables: x and y to pass them as inputs to the function f.
return grad: Returns the gradient vector as a list of partial derivatives.

This general approach allows to calculate the gradient of any function with any amount of variables.