Gradient Descent Fundamentals

Introduction

Welcome to the second lesson of "Training Neural Networks: The Backpropagation Algorithm"! In our previous lesson, we introduced the concept of loss functions and focused specifically on mean squared error (MSE) as a way to measure how far our neural network's predictions deviate from the ground truth. We learned that a loss function acts as a report card for our model's performance, with lower values indicating better predictions. But a critical question remains: how do we actually use this loss function to improve our model? This is where gradient descent comes in. Gradient descent is the fundamental optimization algorithm that powers most neural network training. It provides a systematic way to adjust the model's weights to minimize the loss function. Today, we'll build an intuitive understanding of this powerful algorithm and implement a simple example to see it in action.

Understanding Optimization Landscapes

Let's start by considering the broader optimization problem we're trying to solve when training neural networks. When training a neural network, our goal is to find the set of weights that minimize the loss function. We can visualize this as finding the lowest point in a landscape where: The landscape represents our loss function. The coordinates on this landscape represent our model parameters (weights). The height at each point represents the loss value. Our objective is to find the lowest point (global minimum) in this landscape. For simple problems, we could try to solve this mathematically by setting the derivative of the loss function with respect to each weight to zero and solving the resulting equations. However, for neural networks with thousands or millions of parameters, this approach is computationally impossible. Instead, we need an iterative algorithm that can gradually move toward the minimum. This is precisely what gradient descent does — it starts at some point on the loss landscape and takes steps in the direction that leads downhill most quickly.

Visualizing a Loss Landscape

w_1

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing 1D Gradient Descent: The Algorithm

Now, let's implement the gradient descent algorithm to find this minimum:

C++

int main() {
public:
    double current_x = 0.0;  // Initial guess
    double learning_rate = 0.1;
    int epochs = 30;

    std::cout << "Minimizing f(x) = x^2 - 4x + 5. Starting at x = " 
              << std::fixed << std::setprecision(2) << current_x << std::endl;

    for (int i = 0; i < epochs; i++) {
        // Calculate the gradient at the current position
        double gradient = derivative_simple_quadratic(current_x);
        
        // Update the position by moving in the opposite direction of the gradient
        current_x = current_x - learning_rate * gradient;
        
        // Calculate the new function value
        double loss = simple_quadratic_function(current_x);
        
        // Print progress at specified intervals
        if ((i + 1) % 5 == 0 || i == 0) {  // Print every 5 epochs or the first one
            std::cout << "Epoch " << std::setw(2) << (i + 1) 
                      << ": x = " << std::setw(6) << std::setprecision(3) << current_x
                      << ", f(x) = " << std::setw(6) << std::setprecision(3) << loss
                      << ", gradient = " << std::setw(6) << std::setprecision(3) << gradient
                      << std::endl;
        }
    }

    std::cout << std::endl;
    std::cout << "After " << epochs << " epochs, estimated minimum x = " 
              << std::setprecision(3) << current_x 
              << ", f(x) = " << std::setprecision(3) << simple_quadratic_function(current_x) << std::endl;
    std::cout << "The true minimum is at x = 2.0, f(x) = 1.0" << std::endl;

    return 0;
}

int main() {
public:
    double current_x = 0.0;  // Initial guess
    double learning_rate = 0.1;
    int epochs = 30;

    std::cout << "Minimizing f(x) = x^2 - 4x + 5. Starting at x = " 
              << std::fixed << std::setprecision(2) << current_x << std::endl;

    for (int i = 0; i < epochs; i++) {
        // Calculate the gradient at the current position
        double gradient = derivative_simple_quadratic(current_x);
        
        // Update the position by moving in the opposite direction of the gradient
        current_x = current_x - learning_rate * gradient;
        
        // Calculate the new function value
        double loss = simple_quadratic_function(current_x);
        
        // Print progress at specified intervals
        if ((i + 1) % 5 == 0 || i == 0) {  // Print every 5 epochs or the first one
            std::cout << "Epoch " << std::setw(2) << (i + 1) 
                      << ": x = " << std::setw(6) << std::setprecision(3) << current_x
                      << ", f(x) = " << std::setw(6) << std::setprecision(3) << loss
                      << ", gradient = " << std::setw(6) << std::setprecision(3) << gradient
                      << std::endl;
        }
    }

    std::cout << std::endl;
    std::cout << "After " << epochs << " epochs, estimated minimum x = " 
              << std::setprecision(3) << current_x 
              << ", f(x) = " << std::setprecision(3) << simple_quadratic_function(current_x) << std::endl;
    std::cout << "The true minimum is at x = 2.0, f(x) = 1.0" << std::endl;

    return 0;
}

Let's break down what's happening in this implementation:

We start at an initial position (x=0) — this is like placing ourselves randomly on the hill.
For each epoch (iteration):
- Calculate the gradient (slope) at our current position.
- Update our position by moving in the opposite direction of the gradient, scaled by the learning rate.
- Calculate the new function value (loss) to track our progress.
- Print our status at regular intervals to visualize the optimization journey.

This simple loop captures the essence of gradient descent, whether in 1D or in the much higher-dimensional space of neural network weights.

Visualizing the Optimization Journey

When we run our gradient descent implementation, we see the following output:

plaintext

Minimizing f(x) = x^2 - 4x + 5. Starting at x = 0.00
Epoch  1: x =  0.400, f(x) =  3.560, gradient = -4.000
Epoch  5: x =  1.345, f(x) =  1.429, gradient = -1.638
Epoch 10: x =  1.785, f(x) =  1.046, gradient = -0.537
Epoch 15: x =  1.930, f(x) =  1.005, gradient = -0.176
Epoch 20: x =  1.977, f(x) =  1.001, gradient = -0.058
Epoch 25: x =  1.992, f(x) =  1.000, gradient = -0.019
Epoch 30: x =  1.998, f(x) =  1.000, gradient = -0.006

After 30 epochs, estimated minimum x = 1.998, f(x) = 1.000
The true minimum is at x = 2.0, f(x) = 1.0

Minimizing f(x) = x^2 - 4x + 5. Starting at x = 0.00
Epoch  1: x =  0.400, f(x) =  3.560, gradient = -4.000
Epoch  5: x =  1.345, f(x) =  1.429, gradient = -1.638
Epoch 10: x =  1.785, f(x) =  1.046, gradient = -0.537
Epoch 15: x =  1.930, f(x) =  1.005, gradient = -0.176
Epoch 20: x =  1.977, f(x) =  1.001, gradient = -0.058
Epoch 25: x =  1.992, f(x) =  1.000, gradient = -0.019
Epoch 30: x =  1.998, f(x) =  1.000, gradient = -0.006

After 30 epochs, estimated minimum x = 1.998, f(x) = 1.000
The true minimum is at x = 2.0, f(x) = 1.0

This output reveals several important patterns about how gradient descent works:

We start at x=0 with a steep negative gradient (-4.000), indicating we should move to the right.
In the first update, we jump to x=0.400 (0 - 0.1 * (-4)).
As we continue making updates, the size of each step gets smaller as the gradient approaches zero.
By epoch 30, we've reached x=1.998, extremely close to the true minimum at x=2.0.
Notice how the gradient approaches zero as we get closer to the minimum.

We can observe key characteristics of gradient descent:

Convergence pattern: Our progress is rapid at first and then slows as we approach the minimum.
Gradient magnitude: The gradient naturally gets smaller as we get closer to the minimum.
Approximation: We get very close to, but not exactly at, the true minimum with a finite number of iterations.

This pattern of rapid initial progress followed by slower refinement is characteristic of gradient descent and will be similar when training neural networks with millions of parameters.

#include <iostream> #include <iomanip> #include <cmath> // Define our target function: f(x) = x^2 - 4x + 5 double simple_quadratic_function(double x) { return x*x - 4*x + 5; // Minimum at x=2, f(x)=1 } // Define the derivative (gradient) of our function double derivative_simple_quadratic(double x) { return 2*x - 4; }

Introduction

Understanding Optimization Landscapes

Visualizing a Loss Landscape

Join the 1M+ learners on CodeSignal

Gradient Descent: The Intuition

A Simple 1D Illustration of Gradient Descent

The Role of Learning Rate

Implementing 1D Gradient Descent: The Setup

Implementing 1D Gradient Descent: The Algorithm

Visualizing the Optimization Journey

Gradient Descent in Neural Networks

Conclusion and Next Steps